Articulated structured light based-laparoscope

ABSTRACT

In a method of using a structured-light based system, real-time 2D images of a portion of a field of view are captured using an endoscope. A portion of an object in the field of view is illuminated with a structured light pattern, and light reflected from the field of view is detected. From the reflected light, a 3D image of the field of view is constructed, and 3D locations of points on a surface of the object are determined. The real time 3D spatial position of the endoscope and/or a surgical tool is determined. If a distance between the surface the endoscope and/or surgical tool, as determined using the 3D spatial position, falls below a predetermined distance, an alert is generated to notify a user.

FIELD OF THE INVENTION

The present invention generally pertains to a system and method forproviding a 3D laparoscopic image.

BACKGROUND OF THE INVENTION

Laparoscopic surgery is becoming increasingly popular with patientsbecause the scars are smaller and their period of recovery is shorter.Laparoscopic surgery requires special training of the surgeon orgynecologist and the theatre nursing staff. The equipment is oftenexpensive and is not available in all hospitals. During laparoscopicsurgery it is often required to shift the spatial placement of theendoscope in order to present the surgeon with the optimal view.Conventional laparoscopic surgery makes use of either human assistantsthat manually shift the instrumentation or alternatively roboticautomated assistants (such as JP patent No. 06063003).

In laparoscopic surgery, the surgeon performs the operation throughsmall holes using long instruments and observing the internal anatomywith an endoscope camera. The surgeon's performance is largely dependenton the camera position relative to the instruments and on a stable imageshown at the monitor. In general, the surgeon needs a close-up view ofthe area in which he wants to work, however, there are times when anoverview of a large portion of the working area, such as an overall viewof the interior of the abdomen, is desirable.

In addition, conventional laparoscopes provide the surgeon with a 2Dimage of the field of view.

U.S. Pat. No. 6,387,044 discloses a laparoscope apparatus for use inlaparoscopic surgery or the like comprises a cannula with an innerhollow having therein a light guide for introducing light to its tip endfor illuminating the object to be observed, and an endoscope which iscapable of being pulled into and out from the inner hollow of thecannula. The cannula is airtightly sealed with a transparent member atits tip end, while the endoscope has therein an image pick-up with awide-angle lens, and the image pick-up is housed in the cannula close tothe transparent member.

However, U.S. Pat. No. 6,387,044 does not disclose providing a 3D image.

Korean Patent Application KR2009/012326 discloses a laparoscope and animage processing system using the same to efficiently perform alaparoscope operation by outputting an image of the inside of theabdominal cavity. The laparoscope includes a wide angle lens, an opticalfiber, an inserting part, and an optical interface part. The wide anglelens is arranged at one end of the laparoscope, and is a fisheye lens.The focal distance of the wide angle lens is controllable. The opticalfiber delivers incident light from the wide angle lens to the camerapart, and is received in an inner space of the inserting part. Theinserting part is inserted inside the abdominal cavity. The opticalinterface part delivers the light delivered from the optical fiber tothe camera part. The camera part is a two dimensional camera or a threedimensional camera.

However, Korean Patent Application KR2009/012326 requires a 3D camera inorder to provide a 3D image.

U.S. Patent Application US2011069160 discloses an imaging system andmethod of application, including lens designs tailored to be used withparticular transformation algorithms, electronic hardware and algorithmsfor image transformations is described. Exemplary application of thesystem including automotive, photographic and medical endoscopic arealso described. The system enables improved image view and allowscustomization of views by the end user even after installation of theimage system hardware. In U.S. Patent Application US2011069160,mathematical algorithms are used to alter the image, so that acontinuously varying distortion of the image is possible, so thatdesired portions of the image are magnified, but the whole of the imageremains within the field of view.

However, US Patent Application US2011069160 does not disclose providinga 3D image.

Chinese Utility Model CN203042209 discloses a laparoscopic puncturedevice with an intra-cavity full-view auxiliary lens, which relates to asurgical instrument device, in particular to the laparoscopic puncturedevice with the intra-cavity full-view auxiliary lens for abdominalcavity minimally-invasive surgery. A main lens of a laparoscope entersinto the abdominal cavity through a cannular puncture device, thepuncture device is provided with an auxiliary lens which can be used forobserving intra-cavity full view, and the auxiliary lens is a wide-anglelens and cold light source combination body; the auxiliary lens isarranged at the middle part of the puncture device entering into theabdominal cavity, and the auxiliary lens is positioned at the highestposition of the abdominal cavity after the abdominal cavity is subjectedto air inflation; the outer ring of the puncture device is provided witha groove, the wide-angle lens and cold light source combination body ofthe auxiliary lens is embedded in the groove and is combined into awhole with the puncture device; and the wide angle of a wide-angle lensof the auxiliary lens is larger than or equal to 270 degrees, and theresolution ratio of the wide-angle lens of the auxiliary lens is 1-3million pixels. By adding a cold light source and the wide-angle lens,the full-view visible function is added on the basis of the originalfunction of the puncture device, 80 percent of the abdominal cavity canbe within a visible range, motion of all instruments is within thevisible range, the surgical blind area of a surgeon can be removed, andthe surgery can be safer.

However, US Patent Application US2011069160 does not disclose providinga 3D image.

U.S. Patent Application US2002/0097332 discloses a system for achievingperspective-corrected views at a location removed from the site of thecreation of a distorted wide angle image without the transmission ofcontrol signals to the site of image creation. angles of tilt, pan androtation, as well as degrees of magnification, are achieved without theuse of mechanical devices by using transform algorithms. The systemprovides for the transmission of signals related to an uncorrected imagefrom a site where this distorted image is created, with the transmittedsignals being received at one or more processing sites for creating theperspectively-corrected views. Transmission can be via telephone lines,with the system including signal compression and decompression units.wireless transmission is also utilized where desired.

However, US Patent Application US2002/0097332 does not discloseproviding a 3D image.

It is therefore a long felt need to provide an enhanced 3D laparoscopicimage using an articulated endoscope.

SUMMARY OF THE INVENTION

It is an object of the present invention to disclose a system forproviding an enhanced 3D laparoscopic image using an articulatedendoscope.

It is another object of the present invention to disclose astructured-light based endoscope for providing a 3D image of at leastone object within a field of view within a body cavity, comprising:

-   a. at least one endoscope having at least one lens in the    endoscope's distal end;-   b. at least one camera located in the endoscope's proximal end,    configured to real-time provide at least one 2D image of at least a    portion of said field of view by means of said at least one lens;-   c. at least one light source, configured to real-time illuminate at    least a portion of said at least one object within at least a    portion of said field of view with at least one time and space    varying predetermined light pattern;-   d. at least one sensor configured to detect light reflected from    said field of view; and,-   e. a computer program which, when executed by data processing    apparatus, is configured to generate a 3D image of said field of    view;-   f.    wherein said predetermined light pattern is a structured light    pattern; wherein said 3D image is constructable from said detected    light reflected from said field of view and said structured light    pattern.

It is another object of the present invention to disclose the system asdescribed above, additionally comprising at least one maneuveringsystem, configured to maneuver said endoscope in at least two degrees offreedom (DOF).

It is another object of the present invention to disclose the system asdescribed above, wherein said endoscope is an articulating endoscope.

It is another object of the present invention to disclose the system,wherein said construction of said 3D image is by means of calculatingthe world coordinates of at least one point on said at least one object.

It is another object of the present invention to disclose the system,wherein said world coordinates of at least one point on said at leastone object can be calculated from the following equation:

${\overset{\sim}{\alpha} = \frac{n^{T}{\overset{\sim}{x}}_{p}}{n^{T}R_{p}v_{c}}},$

where n^(T) is the transpose of the normal to the plane defined by thestripe id x_(p), {tilde over (x)}_(p)=x_(p)+[δx_(p), 0, f_(p)]^(T) isthe perturbed stripe id x_(p), R_(p) is the rotation matrix defining thetransformation between the world coordinate system and the projectorcoordinate system and v_(c) is the direction of the ray between thestripe id and the object point.

It is another object of the present invention to disclose the system,wherein, for any point X_(w) in world coordinate system, the coordinateX_(c) of the same point in the camera coordinate system is calculatedaccording to the following equation:

X _(c) =C _(c) X _(w),

-   -   where C_(c), the camera perspective projection matrix, is of the        form

$C_{c} = {{{\alpha\begin{bmatrix}f_{x} & {kf}_{y} & x^{0} \\0 & f_{y} & y_{c}^{0} \\0 & 0 & 1\end{bmatrix}}\left\lbrack {R_{c}t_{c}} \right\rbrack}.}$

where α is a proportion coefficient, f_(x) and f_(y) are the camerafocal length scaled to each of the camera image dimensions, k is theshear of the camera coordinate system, x_(c) ⁰ and y_(c) ⁰ are theorigin of X_(c) in image coordinates, and R_(c) and t_(c) define thetransformation between the world coordinate system and the lightsource's coordinate system, with R_(c) being a rotation matrix and t_(c)a translation matrix.

It is another object of the present invention to disclose the system,wherein x_(p) ⁰ is the x-coordinate of the intersection of the opticalaxis and the projector.

It is another object of the present invention to disclose the system,wherein, for any point X_(w) in world coordinate system, the coordinateX_(p) of the same point in the light source coordinate system iscalculated according to the following equation:

X _(p) =C _(p) X _(w),

-   -   where C_(p), the light source perspective projection matrix, is        of the form

$C_{p} = {{\alpha\begin{bmatrix}f_{p} & 0 & x_{p}^{0} \\0 & 0 & 1\end{bmatrix}}\left\lbrack {R_{p}t_{p}} \right\rbrack}$

where α is a proportion coefficient, f_(p) is the light source focallength scaled to projector dimensions, x_(p) ⁰ is the origin of X_(p) inprojector coordinates, and R_(p) and t_(p) define the transformationbetween the world coordinate system and the light source's coordinatesystem, with R_(p) being a rotation matrix and t_(p) a translationmatrix.

It is another object of the present invention to disclose the system,wherein the world coordinates p_(w) of a point P is calculated accordingto the following equation:

${\begin{pmatrix}p_{p} \\p_{s}\end{pmatrix} - \begin{pmatrix}{F_{c}\left( {p_{w};\Theta_{c}} \right)} \\{F_{p}\left( {p_{w};\Theta_{p}} \right)}\end{pmatrix}} = 0$

where p_(p)=(x_(p),y_(p))^(t) is the pixel coordinate of said point,p_(x)=(x_(s)) is the stripe value of said point P, F_(c)(P_(w);Θ_(c))=P_(p)−ϵ_(p) is the noise-free value of the vector of pixelcoordinates, where P_(p) is the vector of measured pixel coordinates andϵ_(p) is the vector of errors in the pixel coordinates; F_(p)(P_(w);Θ_(p))=P_(s)−ϵ_(s) is the noise-free value of the vector of stripecoordinates, where P_(s) is the vector of measured stripe coordinatesand ϵ_(s) is the vector of errors in the stripe coordinates.

It is another object of the present invention to disclose the system,wherein the world coordinates p_(w) of a point P is estimated accordingto the following non-linear least squares (NLLS) equations:

${\min\limits_{\Theta_{c}}{{P_{p} - {F_{c}\left( {P_{w};\Theta_{c}} \right)}}}^{2}}{\min\limits_{\Theta_{p}}{{P_{s} - {F_{p}\left( {P_{w};\Theta_{p}} \right)}}}^{2}}$

It is another object of the present invention to disclose the system,wherein said NLLS equations are solvable by means of a NLLS solvingalgorithm selected from a group consisting of the Gauss-Newtontechnique, the quasi-Newton technique, and the Levenberg-Marquardttechnique.

It is another object of the present invention to disclose the system,wherein the location, in world coordinates, of a kth point on the objectis calculated according to the following equation:

$p_{w}^{k} = \frac{C_{1,2,1}^{k} - {x_{p}C_{3,2,1}^{k}} - {y_{p}C_{1,3,1}^{k}} - {x_{s}C_{1,2,2}^{k}} + {x_{s}x_{p}C_{3,2,2}^{k}} + {x_{s}y_{p}C_{1,3,2}^{k}}}{C_{1,2,1}^{4} - {x_{p}C_{3,2,1}^{4}} - {y_{p}C_{1,3,1}^{4}} - {x_{s}C_{1,2,2}^{4}} + {x_{s}x_{p}C_{3,2,2}^{4}} + {x_{s}y_{p}C_{1,3,2}^{4}}}$

where C_(i,j,l) ^(k)=det (C_(c) ^(i), C_(o) ^(j), C_(p) ^(l), e_(k)) areconstants which depend only on a camera perspective transformationmatrix and a projector perspective transformation matrix.

It is another object of the present invention to disclose the system,additionally comprising a calibration object.

It is another object of the present invention to disclose the system,wherein said calibration object is of predetermined shape and size.

It is another object of the present invention to disclose the system,wherein said calibration object comprises fiducial locations ofpredetermined position.

It is another object of the present invention to disclose the system,wherein said lens is a wide-angle lens.

It is another object of the present invention to disclose the system,wherein said wide-angle lens is selected from a group consisting of afisheye lens, an omnidirectional lens and any combination thereof.

It is another object of the present invention to disclose the system,wherein said structured light uses at least one of a group consistingof: temporal sequencing, spatial sequencing, wavelength sequencing andany combination thereof.

It is another object of the present invention to disclose the system,wherein cancerous tissue can be differentiated from normal tissue bymeans of said wavelength sequencing.

It is another object of the present invention to disclose the system,wherein the relationship between the location of a point in said cameraimage, the location of a point in a light source and the location of apoint in space is known for all said points in said camera image, saidpoints in said light source and said points in space.

It is another object of the present invention to disclose the system,wherein said known relationships are used to generate said 3D image fromsaid 2D camera image.

It is another object of the present invention to disclose the system,wherein said system comprises at least one location estimating meansconfigured to real-time locate the 3D spatial position at any given timet of a member of a group consisting of: said endoscope, a surgical tooland any combination thereof.

It is another object of the present invention to disclose the system,wherein said at least one location estimating means comprises (a) atleast one element selected from a group consisting of optical imagingmeans, radio frequency transmitting and receiving means, at least onemark on said at least one surgical tool and any combination thereof and,(b) at least one surgical instrument spatial location softwareconfigured to estimate said 3D spatial position of said at least onesurgical tool by means of said element.

It is another object of the present invention to disclose the system,wherein said at least one location estimating means is an interfacesubsystem between a surgeon and said at least one surgical tool, theinterface subsystem comprising:

-   a. at least one array comprising N regular or patterned light    sources, where N is a positive integer;-   b. at least one array comprising M cameras, each of the M cameras,    where M is a positive integer;-   c. optional optical markers and means for attaching the optical    marker to the at least one surgical tool; and;-   d. a computerized algorithm operable via the controller, the    computerized algorithm configured to record images received by each    camera of each of the M cameras and to calculate therefrom the    position of each of the tools, and further configured to provide    automatically the results of the calculation to the human operator    of the interface.

It is another object of the present invention to disclose the system,wherein display of said 3D image is by means of a member of a groupconsisting of: a 3D screen, 3D glasses, autostereoscopy, holography, avolumetric display, integral imaging, wiggle stereoscopy, an activeshutter 3d system, a polarized 3d system, an interference filter system,a color anaglyph systems a Chromadepth system, an over/under format, andany combination thereof.

It is another object of the present invention to disclose the system,wherein said tools are detectable by means of depth cues in said 3Dimage.

It is another object of the present invention to disclose the system,wherein at least one structure selected from a group consisting of anorgan, a blood vessel, a nerve, a limb and any combination thereof isdetectable by said system.

It is another object of the present invention to disclose the system,wherein at least one said structure are emphasizable by a member of agroup consisting of: virtual reality overlays, color markings, patternmarkings, outlines, arrows, and any combination thereof.

It is another object of the present invention to disclose the system,wherein data from at least one other imaging modality is storable in adatabase.

It is another object of the present invention to disclose the system,wherein said data from said at least one other imaging modality ismatchable to and alignable with said 3D image, such that substantiallyall information from said at least one other imaging modality and said3D image is viewable as a single 3D image.

It is another object of the present invention to disclose the system,wherein said at least one other imaging modality is selected from agroup consisting of: CT, MRI, ultrasound and PET and any combinationthereof.

It is another object of the present invention to disclose the system,wherein additional information displayable by said system is selectedfrom a group consisting of: recommendations useful in directing theprogress of an operation, information useful in assisting an operator inmaking a decision, information useful for a record of the operation andany combination thereof.

It is another object of the present invention to disclose the system,wherein said information is displayable visually or audibly.

It is another object of the present invention to disclose the system,wherein said information is selected from a group consisting of: camerapose, body temperature, time, elapsed time since start of operation,patient notes, notes made during the

It is another object of the present invention to disclose the system,additionally comprising at least one joystick unit in communication withsaid maneuvering system, configured to operate said maneuvering system.

It is another object of the present invention to disclose the system,wherein operation of said joystick results in movement of said endoscopeby means of said maneuvering system.

It is another object of the present invention to disclose the system,wherein said joystick unit is wearable by a user of said system.

It is another object of the present invention to disclose the system,wherein said joystick unit is coupled to at least one surgical tool usedin said medical procedure.

It is another object of the present invention to disclose the system,wherein said at least one surgical tool is said endoscope.

It is another object of the present invention to disclose the system,wherein said movement of said joystick is proportional to said movementof said endoscope.

It is another object of the present invention to disclose the system,wherein, if said joystick unit's speed of motion is above apredetermined value, said endoscope's speed is at a predetermined value.

It is another object of the present invention to disclose the system,wherein said joystick unit is a force joystick.

It is another object of the present invention to disclose the system,wherein said joystick unit comprises a base and lever coupled to saidbase, such that movement of said lever results in movement of saidendoscope; further wherein said movement of said lever is proportionalto said movement of said endoscope.

It is another object of the present invention to disclose the system,wherein said joystick unit comprises a base and a button jointlyconnected to said base, such that movement of said button results inmovement of said endoscope; further wherein said movement of said buttonis proportional to said movement of said endoscope.

It is another object of the present invention to disclose the system,wherein said joystick unit comprises a touchscreen, such that a touchand a movement on said touchscreen results in movement of saidendoscope; further wherein said touch and movement on said touchscreenis proportional to said movement of said endoscope.

It is another object of the present invention to disclose the system,wherein said location within said surgical environment of said humanbody is determinable from pressure on a portion of said touchscreen.

It is another object of the present invention to disclose the system,wherein said portion of said touchscreen is that which displays an imageof said location.

It is another object of the present invention to disclose the system,additionally comprising means configured to restrain the velocity ofsaid endoscope, such that when said means are activated, the velocity ofsaid endoscope is restrained.

It is another object of the present invention to disclose the system,wherein said joystick unit additionally comprises n sensors, where n isan integer larger than one.

It is another object of the present invention to disclose the system,wherein at least one of said n sensors is activated in case of powerfailure.

It is another object of the present invention to disclose the system,wherein at least one of said n sensors is activated when said system isconnected to power.

It is another object of the present invention to disclose the system,wherein said sensors are selected from a group consisting of a motionsensor, a heat sensor, an electric sensor, a sound sensor, a pressuresensor, an optical sensor and any combination thereof.

It is another object of the present invention to disclose the system,wherein said joystick unit is characterized by an external surface.

It is another object of the present invention to disclose the system,wherein said at least one motion sensor detects motion upon saidexternal surface.

It is another object of the present invention to disclose the system,wherein said at least one motion sensor detects motion perpendicular tosaid external surface.

It is another object of the present invention to disclose the system,wherein said at least one heat sensor is configured to sensetemperatures in the range of about 35 to about 42 degrees.

It is another object of the present invention to disclose the system,wherein said system is configured to enable maneuvering of saidendoscope at such times as said at least one heat sensor sensestemperatures in the range of about 35 to about 42 degrees.

It is another object of the present invention to disclose the system,wherein said at least one heat sensor is configured to provide thermalimage; said at least one heat sensor is coupled to a processing unitcomprising instructions configured, when executed, to provide saidsystem user with said thermal image.

It is another object of the present invention to disclose the system,wherein said system is configured to enable maneuvering of saidendoscope at such times as analysis of said image by said processingunit detects the image of a human hand; further wherein said system isconfigured to prevent maneuvering of said endoscope at such times whensaid analysis of said image by said processing unit fails to detect animage of a human hand.

It is another object of the present invention to disclose the system,wherein said at least one electric sensor is configured to sense powerfailure.

It is another object of the present invention to disclose the system,wherein said at least one electric sensor is configured to senseelectric conductivity of a human body.

It is another object of the present invention to disclose the system,wherein said system is configured to enable maneuvering of saidendoscope at such times when said sensor senses the conductivity of saidsubject's body; further wherein said system is configured to preventmaneuvering of said endoscope at such times as said sensor fails tosense the conductivity of said subject's body.

It is another object of the present invention to disclose the system,wherein said at least one sound sensor is configured to sense at leastone predetermined sound pattern.

It is another object of the present invention to disclose the system,wherein said endoscope is maneuverable according to said at least onepredetermined sound pattern sensed by said at least one sound sensor.

It is another object of the present invention to disclose the system,wherein said at least one pressure sensor is configured to sensepressure applied to said joystick unit.

It is another object of the present invention to disclose the system,wherein said pressure sensed by said at least one pressure sensoraffects the system for maneuvering an endoscope (SFME) in a mannerselected from a group consisting of: when said pressure sensed by saidat least one pressure sensor is above a predetermined value, said SFMEis activated; when said pressure sensed by said at least one pressuresensor is above a predetermined value, said SFME is de-activated; andwhen said pressure sensed by said at least one pressure sensor is belowa predetermined value, said SFME is de-activated.

It is another object of the present invention to disclose the system,wherein said at least one optical sensor is configured to sense visualchanges according to at least one predetermined visual pattern.

It is another object of the present invention to disclose the system,wherein said endoscope is maneuverable according to said at least onepredetermined visual pattern.

It is another object of the present invention to disclose the system,additionally comprising an interface system configured to enablecommunication between said joystick unit and said maneuvering system.

It is another object of the present invention to disclose the system,wherein said communication means comprises a member selected from agroup consisting of a wired communication means, a wirelesscommunication means and any combination thereof.

It is another object of the present invention to disclose the system,wherein said SFME comprises at least one second joystick unit configuredto zoom the endoscope by means of said maneuvering system.

It is another object of the present invention to disclose the system,wherein a single device comprises said joystick unit and said secondjoystick unit.

It is another object of the present invention to disclose the system,wherein said second joystick unit is wearable by said system user.

It is another object of the present invention to disclose the system,wherein said second joystick unit is coupled to at least one surgicaltool.

It is another object of the present invention to disclose the system,wherein said at least one surgical tool is said endoscope.

It is another object of the present invention to disclose the system,wherein said at least one joystick unit is configured to control and todirect said endoscope via said maneuvering system on a surgical tool.

It is another object of the present invention to disclose the system,wherein selection of said at least one surgical tool is obtained byactivating said at least one joystick unit; further wherein theactivation of said at least one joystick unit is obtained by depressionof said joystick unit, voice activating the same, prolonged depressionon the same, double clicking on the same and any combination thereof.

It is another object of the present invention to disclose the system,wherein said endoscope comprises at least one proximity sensorpositioned on an outer circumference of the same.

It is another object of the present invention to disclose the system,additionally comprising a set of instructions which, when executed bydata processing apparatus, are configured to control said maneuvering ofsaid endoscope.

It is another object of the present invention to disclose the system,wherein said instructions comprise a predetermined set of rules selectedfrom a group consisting of: most used tool rule, right tool rule, lefttool rule, field of view rule, no fly zone rule, a route rule,environmental rule, operator input rule, proximity rule; collisionprevention rule, history-based rule, tool-dependent ALLOWED andRESTRICTED movements rule, preferred volume zone rule, preferred toolrule, movement detection rule, tagged tool rule, change of speed ruleand any combination thereof.

It is another object of the present invention to disclose the system,wherein said route rule comprises a communicable database storingpredefined route in which said at least one surgical tool is configuredto move within said surgical environment; said predefined routecomprises n 3D spatial positions of said at least one surgical tool; nis an integer greater than or equal to 2; said ALLOWED movements aremovements in which said at least one surgical tool is locatedsubstantially in at least one of said n 3D spatial positions of saidpredefined route, and said RESTRICTED movements are movements in whichsaid location of said at least one surgical tool is substantiallydifferent from said n 3D spatial positions of said predefined route.

It is another object of the present invention to disclose the system,wherein said environmental rule comprises a comprises a communicabledatabase; said communicable database configured to receive at least onereal-time image of said surgical environment and is configured toperform real-time image processing of the same and to determine the 3Dspatial position of hazards or obstacles in said surgical environment;said environmental rule is configured to determine said ALLOWED andRESTRICTED movements according to said hazards or obstacles in saidsurgical environment, such that said RESTRICTED movements are movementsin which said at least one surgical tool is located substantially in atleast one of said 3D spatial positions, and said ALLOWED movements aremovements in which the location of said at least one surgical tool issubstantially different from said 3D spatial positions.

It is another object of the present invention to disclose the system,wherein said hazards or obstacles in said surgical environment areselected from a group consisting of tissue, a surgical tool, an organ,an endoscope and any combination thereof.

It is another object of the present invention to disclose the system,wherein said operator input rule comprises a communicable database; saidcommunicable database is configured to receive an input from theoperator of said system regarding said ALLOWED and RESTRICTED movementsof said at least one surgical tool.

It is another object of the present invention to disclose the system,wherein said input comprises n 3D spatial positions; n is an integergreater than or equal to 2; wherein at least one of which is defined asALLOWED location and at least one of which is defined as RESTRICTEDlocation, such that said ALLOWED movements are movements in which saidat least one surgical tool is located substantially in at least one ofsaid n 3D spatial positions, and said RESTRICTED movements are movementsin which the location of said at least one surgical tool issubstantially different from said n 3D spatial positions.

It is another object of the present invention to disclose the system,wherein said input comprises at least one rule according to whichALLOWED and RESTRICTED movements of said at least one surgical tool aredetermined, such that the spatial position of said at least one surgicaltool is controlled by said controller according to said ALLOWED andRESTRICTED movements.

It is another object of the present invention to disclose the system,wherein said predetermined set of rules comprises at least one ruleselected from a group consisting of: most used tool, right tool rule,left tool rule, field of view rule, no fly zone rule, route rule,environmental rule, operator input rule, proximity rule, collisionprevention rule, preferred volume zone rule, preferred tool rule,movement detection rule, history-based rule, tool-dependent ALLOWED andRESTRICTED movements rule, and any combination thereof.

It is another object of the present invention to disclose the system,wherein said operator input rule converts an ALLOWED movement to aRESTRICTED movement and a RESTRICTED movement to an ALLOWED movement.

It is another object of the present invention to disclose the system,wherein said proximity rule is configured to define a predetermineddistance between at least two surgical tools; said ALLOWED movements aremovements which are within the range or out of the range of saidpredetermined distance, and said RESTRICTED movements are movementswhich are out of the range or within the range of said predetermineddistance.

It is another object of the present invention to disclose the system,wherein said proximity rule is configured to define a predeterminedangle between at least three surgical tools; said ALLOWED movements aremovements which are within the range or out of the range of saidpredetermined angle, and said RESTRICTED movements which are out of therange or within the range of said predetermined angle.

It is another object of the present invention to disclose the system,wherein said collision prevention rule is configured to define apredetermined distance between said at least one surgical tool and ananatomical element within said surgical environment; said ALLOWEDmovements are movements which are in a range that is larger than saidpredetermined distance, and said RESTRICTED movements are movementswhich is in a range that is smaller than said predetermined distance.

It is another object of the present invention to disclose the system,wherein said anatomical element is selected from a group consisting oftissue, organ, another surgical tool and any combination thereof.

It is another object of the present invention to disclose the system,wherein said right tool rule is configured to determine said ALLOWEDmovement of said endoscope according to the movement of the surgicaltool positioned to right of said endoscope; further wherein said lefttool rule is configured to determine said ALLOWED movement of saidendoscope according to the movement of the surgical tool positioned toleft of said endoscope.

It is another object of the present invention to disclose the system,wherein said tagged tool rule comprises means configured to tag at leastone surgical tool within said surgical environment and to determine saidALLOWED movement of said endoscope so as to constantly track themovement of said tagged surgical tool.

It is another object of the present invention to disclose the system,wherein said field of view rule comprises a communicable databasecomprising n 3D spatial positions; n is an integer greater than or equalto 2; the combination of all of said n 3D spatial positions provides apredetermined field of view; said field of view rule is configured todetermine said ALLOWED movement of said endoscope within said n 3Dspatial positions so as to maintain a constant field of view, such thatsaid ALLOWED movements are movements in which said endoscope is locatedsubstantially in at least one of said n 3D spatial positions, and saidRESTRICTED movements are movements in which the location of saidendoscope is substantially different from said n 3D spatial positions.

It is another object of the present invention to disclose the system,wherein said preferred volume zone rule comprises a communicabledatabase comprising n 3D spatial positions; n is an integer greater thanor equal to 2; said n 3D spatial positions provides said preferredvolume zone; said preferred volume zone rule is configured to determinesaid ALLOWED movement of said endoscope within said n 3D spatialpositions and RESTRICTED movement of said endoscope outside said n 3Dspatial positions, such that said ALLOWED movements are movements inwhich said endoscope is located substantially in at least one of said n3D spatial positions, and said RESTRICTED movements are movements inwhich the location of said endoscope is substantially different fromsaid n 3D spatial positions.

It is another object of the present invention to disclose the system,wherein said preferred tool rule comprises a communicable database, saiddatabase stores a preferred tool; said preferred tool rule is configuredto determine said ALLOWED movement of said endoscope to constantly trackthe movement of said preferred tool.

It is another object of the present invention to disclose the system,wherein said no fly zone rule comprises a communicable databasecomprising n 3D spatial positions; n is an integer greater than or equalto 2; said n 3D spatial positions define a predetermined volume withinsaid surgical environment; said no fly zone rule is configured todetermine said RESTRICTED movement if said movement is within said nofly zone and ALLOWED movement if said movement is outside said no flyzone, such that said RESTRICTED movements are movements in which said atleast one of said surgical tool is located substantially in at least oneof said n 3D spatial positions, and said ALLOWED movements are movementsin which the location of said at least one endoscope is substantiallydifferent from said n 3D spatial positions.

It is another object of the present invention to disclose the system,wherein said most used tool rule comprises a communicable databasecounting the amount of movement of each said surgical tool; said mostused tool rule is configured to constantly position said endoscope totrack the movement of the most moved surgical tool.

It is another object of the present invention to disclose the system,wherein said system further comprises a maneuvering subsystemcommunicable with said controller, said maneuvering subsystem isconfigured to spatially reposition said at least one surgical toolduring a surgery according to said predetermined set of rules; furtherwherein said system is configured to alert the physician of saidRESTRICTED movement of said at least one surgical tool.

It is another object of the present invention to disclose the system,wherein said alert is selected from a group consisting of audiosignaling, voice signaling, light signaling, flashing signaling and anycombination thereof.

It is another object of the present invention to disclose the system,wherein said ALLOWED movement is permitted by said controller and saidRESTRICTED movement is denied by said controller.

It is another object of the present invention to disclose the system,wherein said history-based rule comprises a communicable databasestoring each 3D spatial position of each said surgical tool, such thateach movement of each surgical tool is stored; said history-based ruleis configured to determine said ALLOWED and RESTRICTED movementsaccording to historical movements of said at least one surgical tool,such that said ALLOWED movements are movements in which said at leastone surgical tool is located substantially in at least one of said 3Dspatial positions, and said RESTRICTED movements are movements in whichthe location of said at least one surgical tool is substantiallydifferent from said 3D spatial positions.

It is another object of the present invention to disclose the system,wherein said tool-dependent ALLOWED and RESTRICTED movements rulecomprises a communicable database; said communicable database isconfigured to store predetermined characteristics of at least one ofsaid surgical tool; said tool-dependent ALLOWED and RESTRICTED movementsrule is configured to determine said ALLOWED and RESTRICTED movementsaccording to said predetermined characteristics of said surgical tool;such that ALLOWED movements are movements of said endoscope which tracksaid surgical tool having said predetermined characteristics.

It is another object of the present invention to disclose the system,wherein said predetermined characteristics of said surgical tool areselected from a group consisting of: physical dimensions, structure,weight, sharpness, and any combination thereof.

It is another object of the present invention to disclose the system,wherein said movement detection rule comprises a communicable databasecomprising the real-time 3D spatial positions of each said surgicaltool; said movement detection rule is configured to detect movement ofsaid at least one surgical tool when a change in said 3D spatialpositions is received, such that said ALLOWED movements are movements inwhich said endoscope is re-directed to focus on said moving surgicaltool.

It is another object of the present invention to disclose the system,wherein, for each said instructing function, there exists a weightingfunction, said weighting function, order of execution of saidinstructions selectable by said weighting function.

It is another object of the present invention to disclose the system,further comprising a maneuvering subsystem communicable with saidcontroller, said maneuvering subsystem is configured to spatiallyreposition said at least one surgical tool during a surgery according tosaid predetermined set of rules, such that if said movement of said atleast one surgical tool is a RESTRICTED movement, said maneuveringsubsystem prevents said movement.

It is another object of the present invention to disclose the system,additionally comprising

-   (a) at least one wearable operator comprising at least one wireless    transmitter, configured to transmit a signal once said at least one    wearable operator is activated; said at least one wearable operator    is either wire or wirelessly in communication with at least one    surgical instrument;-   (b) at least one wireless receiver; configured to receive said    signal sent by said transmitter;-   (c) at least one laparoscopy computerized system, in communication    with said wireless receiver, configured to provide a visual onscreen    depiction of said at least one instrument to be selected following    the activation of said at least one wearable operator; and,-   (d) at least one video screen; wherein said system is configured to    control and to direct said endoscope via said laparoscopy    computerized system and said maneuvering system on said instrument    to be selected following the activation of said at least one    wearable operator.

It is another object of the present invention to disclose the system,wherein said communication between said at least one of said wearableoperators and said instrument is either wire or wirelessly coupling.

It is another object of the present invention to disclose the system,wherein said wearable operator is worn by said surgeon on apredetermined body part.

It is another object of the present invention to disclose the system,wherein said predetermined body part is selected from a group consistingof: the hand of said surgeon, at least one of the fingers of saidsurgeon, the thigh of said surgeon, the neck of said surgeon, at leastone of the legs of said surgeon, the knee of said surgeon, the head ofsaid surgeon and any combination thereof.

It is another object of the present invention to disclose the system,wherein the shape of said wearable operator is selected from a groupconsisting of a ring, a bracelet and any combination thereof.

It is another object of the present invention to disclose the system,wherein said wearable operator is coupled to a predetermined location onsaid instrument by means of an adaptor.

It is another object of the present invention to disclose the system,wherein said wearable operator is adjustable so as to fit saidpredetermined location of said different instruments, each of which ischaracterized by a different size and shape.

It is another object of the present invention to disclose the system,wherein said wearable operator comprises a body having at least twoportions at least partially overlapping each other; said two portionsare configured to grasp and hold either said instrument or saidpredetermined body part there-between, such that a tight-fit couplingbetween said two portions and said instrument or said predetermined bodypart is obtained.

It is another object of the present invention to disclose the system,wherein one of said two portions is rotationally movable relative to theother, such that when said wearable operator is coupled to saidinstrument, fine-tuned movement of said two body portions is obtainableso as to provide said tight-fit coupling between said two portions andsaid instrument or said predetermined body part.

It is another object of the present invention to disclose the system,wherein said two portions are rotationally movable relative to eachother, such that when said wearable operator is coupled to saidinstrument, fine-tuned movement of said two body portions is obtainableso as to provide said tight-fit coupling between said two portions andsaid instrument or said predetermined body part.

It is another object of the present invention to disclose the system,wherein said wearable operator comprises (a) at least one flexible andstretchable strip; and (b) loop-closing means configured to close a loopwith said at least one flexible and stretchable strip; said at least oneflexible and stretchable strip and said loop-closing means are providedso as to fit said wearable operator to at least one selected from agroup consisting of (a) said predetermined location of said differentinstruments; (b) said predetermined body part of said surgeon, each ofwhich is characterized by a different size and shape.

It is another object of the present invention to disclose the system,wherein said flexible and stretchable strip is made of material selectedfrom a group consisting of silicone, rubber and any combination thereof.

It is another object of the present invention to disclose the system,wherein said wireless transmitter is configured to locate the positionof at least one of said instruments.

It is another object of the present invention to disclose the system,wherein selection of said at least one instrument is obtained byactivating said at least one wearable operator; further wherein theactivation of said at least one wearable operator is obtained bydepression on a predetermined location in said wearable operator, voiceactivating the same, prolonged depression on the same, double clickingon the same and any combination thereof.

It is another object of the present invention to disclose the system,wherein said laparoscopy computerized system directs said endoscope byusing image information shown on said video screen without said help ofassistants.

It is another object of the present invention to disclose the system,wherein said conventional laparoscopy computerized system comprises atleast one surgical instrument spatial location software, configured tolocate the 3D spatial position of said at least one instrument; furtherwherein said conventional laparoscopy computerized system comprises atleast one automated assistant maneuvering system; said automatedassistant maneuvering system is coupled to said endoscope and isconfigured to direct said endoscope to said at least one instrument,said instrument selected following the activation of said at least onewearable operator.

It is another object of the present invention to disclose the system,wherein each transmitted signal from said wearable operator and saidwireless transmitter is matched to at least one of said instruments.

It is another object of the present invention to disclose the system,wherein said articulating tool has articulations substantially at thetip of said tool, substantially along the body of said too, and anycombination thereof.

It is another object of the present invention to disclose the system,wherein control of articulation is selected from a group consisting ofhardware control, software control and any combination thereof.

It is another object of the present invention to disclose the system,wherein said tool has articulation in a regions selected from a groupconsisting of near the tip of said tool, on the body of said tool, andany combination thereof.

It is another object of the present invention to disclose a method forproviding a 3D image of at least one object within a field of viewwithin a body cavity by means of a structured-light based endoscope,comprising steps of:

-   a. providing a structured-light based endoscope comprising:    -   i. at least one endoscope having at least one lens in the        endoscope's distal end;    -   ii. at least one camera located in the endoscope's proximal end,        configured to real-time provide at least one 2D image of at        least a portion of said field of view by means of said at least        one lens;    -   iii. at least one light source, configured to real-time        illuminate at least a portion of said at least one object within        at least a portion of said field of view with at least one time        and space varying predetermined light pattern;    -   iv. at least one sensor configured to detect light reflected        from said field of view;    -   v. a computer program which, when executed by data processing        apparatus, is configured to generate a 3D image of said field of        view; and,-   b. maneuvering said endoscope and controlling the movements of the    same;-   c. illuminating said at least a portion of said field of view with    said at least one time and space varying predetermined light    pattern;-   d. detecting said light reflected from said field of view;-   e. generating, from said light reflected from said field of view,    said 3D image of said field of view;

wherein said predetermined light pattern is a structured light pattern;further wherein said 3D image is constructable from said detected lightreflected from said field of view and said structured light pattern.

It is another object of the present invention to disclose the method asdescribed above, wherein said system additionally comprising at leastone maneuvering system, configured to maneuver said endoscope in atleast two degrees of freedom (DOF);

It is another object of the present invention to disclose the method asdescribed above, wherein said endoscope is an articulating endoscope.

It is another object of the present invention to disclose the method,additionally comprising step of constructing said 3D image bycalculating the world coordinates of at least one point on said at leastone object.

It is another object of the present invention to disclose the method,additionally comprising step of calculating said world coordinates ofsaid at least one point on said at least one object from the followingequation:

${\overset{\sim}{\alpha} = \frac{n^{T}{\overset{\sim}{x}}_{p}}{n^{T}R_{p}v_{c}}},$

where n^(T) is the transpose of the normal to the plane defined by thestripe id x_(p), {tilde over (x)}_(p)=x_(p)+[δx_(p), 0, f_(p)]^(T) isthe perturbed stripe id x_(p), R_(p) is the rotation matrix defining thetransformation between the world coordinate system and the projectorcoordinate system and v_(c) is the direction of the ray between thestripe id and the object point.

It is another object of the present invention to disclose the method,additionally comprising step of, for any point X_(w) in world coordinatesystem, calculating the coordinate X_(c) of the same point in the cameracoordinate system according to the following equation:

X _(c) =C _(c) X _(w),

where C_(c), the camera perspective projection matrix, is of the form

$C_{c} = {{{\alpha\begin{bmatrix}f_{x} & {kf}_{y} & x_{c}^{0} \\0 & f_{y} & y_{c}^{0} \\0 & 0 & 1\end{bmatrix}}\left\lbrack {R_{c}t_{c}} \right\rbrack}.}$

where α is a proportion coefficient, f_(x) and f_(y) are the camerafocal length scaled to each of the camera image dimensions, k is theshear of the camera coordinate system, x_(c) ⁰ and y_(c) ⁰ are theorigin of X_(c) in image coordinates, and R_(c) and t_(c) define thetransformation between the world coordinate system and the lightsource's coordinate system, with R_(c) being a rotation matrix and t_(c)a translation matrix.

It is another object of the present invention to disclose the method,additionally comprising step of defining x_(p) ⁰ to be the x-coordinateof the intersection of the optical axis and the projector

It is another object of the present invention to disclose the method,additionally comprising step of, for any point X_(w) in world coordinatesystem, calculating the coordinate X_(p) of the same point in the lightsource coordinate system according to the following equation:

X _(p) =C _(p) X _(w),

where C_(p), the light source perspective projection matrix, is of theform

$C_{p} = {{\alpha\begin{bmatrix}f_{p} & 0 & x_{p}^{0} \\0 & 0 & 1\end{bmatrix}}\left\lbrack {R_{p}t_{p}} \right\rbrack}$

where α is a proportion coefficient, f_(p) is the light source focallength scaled to projector dimensions, x_(p) ⁰ is the origin of X_(p) inprojector coordinates, and R_(p) and t_(p) define the transformationbetween the world coordinate system and the light source's coordinatesystem, with R_(p) being a rotation matrix and t_(p) a translationmatrix.

It is another object of the present invention to disclose the method,additionally comprising step of calculating the world coordinates p_(w)of a point P according to the following equation:

${\begin{pmatrix}p_{p} \\p_{s}\end{pmatrix} - \begin{pmatrix}{F_{c}\left( {p_{w};\Theta_{c}} \right)} \\{F_{p}\left( {p_{w};\Theta_{p}} \right)}\end{pmatrix}} = 0$

where p_(p)=(x_(p),y_(p))^(t) is the pixel coordinate of said point,p_(x)=(x_(s)) is the stripe value of said point P, F_(c)(P_(w);Θ_(c))=P_(s)−ϵ_(p) is the noise-free value of the vector of pixelcoordinates, where P_(p) is the vector of measured pixel coordinates andϵ_(p) is the vector of errors in the pixel coordinates; F_(p)(P_(w);ΘP)=P_(s)−ϵ_(s) is the noise-free value of the vector of stripecoordinates, where P_(s) is the vector of measured stripe coordinatesand ϵ_(s) is the vector of errors in the stripe coordinates.

It is another object of the present invention to disclose the method,additionally comprising step of estimating the world coordinates p_(w)of a point P according to the following non-linear least squares (NLLS)equations:

${\min\limits_{\Theta_{c}}{{P_{p} - {F_{c}\left( {P_{w};\Theta_{c}} \right)}}}^{2}}{\min\limits_{\Theta_{p}}{{P_{s} - {F_{p}\left( {P_{w};\Theta_{p}} \right)}}}^{2}}$

It is another object of the present invention to disclose the method,additionally comprising step of solving said NLLS equations using a NLLSsolving algorithm selected from a group consisting of the Gauss-Newtontechnique, the quasi-Newton technique, and the Levenberg-Marquardttechnique.

It is another object of the present invention to disclose the method,additionally comprising a step of calculating the location, in worldcoordinates, of a kth point on the object according to the followingequation

$p_{w}^{k} = \frac{\begin{matrix}{C_{1,2,1}^{k} - {x_{p}C_{3,2,1}^{k}} - {y_{p}C_{1,3,1}^{k}} - {x_{s}C_{1,2,2}^{k}} +} \\{{x_{s}x_{p}C_{3,2,2}^{k}} + {x_{s}y_{p}C_{1,3,2}^{k}}}\end{matrix}}{\begin{matrix}{C_{1,2,1}^{4} - {x_{p}C_{3,2,1}^{4}} - {y_{p}C_{1,3,1}^{4}} - {x_{s}C_{1,2,2}^{4}} +} \\{{x_{s}x_{p}C_{3,2,2}^{4}} + {x_{s}y_{p}C_{1,3,2}^{4}}}\end{matrix}}$

Where (C_(c) ^(i), C_(o) ^(j), C_(p) ^(k), e_(k)) are constants whichdepend only on a camera perspective transformation matrix and aprojector perspective transformation matrix.

It is another object of the present invention to disclose the method,additionally comprising step of providing a calibration object.

It is another object of the present invention to disclose the method,additionally comprising step of providing said calibration object of apredetermined shape and size.

It is another object of the present invention to disclose the method,additionally comprising step of providing said calibration object withfiducial marks at predetermined positions.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said lens to be a wide-anglelens.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said wide-angle lens from agroup consisting of a fisheye lens, an omnidirectional lens and anycombination thereof.

It is another object of the present invention to disclose the method,additionally comprising step of using at least one of a group consistingof: temporal sequencing, spatial sequencing, wavelength sequencing andany combination thereof in said structured light pattern.

It is another object of the present invention to disclose the method,additionally comprising step of differentiating cancerous tissue fromnormal tissue by means of said wavelength sequencing.

It is another object of the present invention to disclose the method,additionally comprising step of determining the relationship between thelocation of a point in said camera image, the location of a point in alight source and the location of a point in space for all said points insaid camera image, said points in said light source and said points inspace.

It is another object of the present invention to disclose the method,additionally comprising step of using said known relationships togenerate said 3D image from said 2D camera image.

It is another object of the present invention to disclose the method,additionally comprising at least one location estimating meansconfigured to real-time locate the 3D spatial position at any given timet of a member of a group consisting of: said endoscope, a surgical tooland any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising steps of providing said at least one locationestimating means comprising (a) at least one element selected from agroup consisting of optical imaging means, radio frequency transmittingand receiving means, at least one mark on said at least one surgicaltool and any combination thereof and, (b) at least one surgicalinstrument spatial location software configured to estimate said 3Dspatial position of said at least one surgical tool by means of saidelement.

It is another object of the present invention to disclose the method,additionally comprising steps of selecting said at least one locationestimating means to be an interface subsystem between a surgeon and saidat least one surgical tool, the interface subsystem comprising:

-   a. at least one array comprising N regular or patterned light    sources, where N is a positive integer;-   b. at least one array comprising M cameras, each of the M cameras,    where M is a positive integer;-   c. optional optical markers and means for attaching the optical    marker to the at least one surgical tool; and;-   d. a computerized algorithm operable via the controller, the    computerized algorithm configured to record images received by each    camera of each of the M cameras and to calculate therefrom the    position of each of the tools, and further configured to provide    automatically the results of the calculation to the human operator    of the interface.

It is another object of the present invention to disclose the method,additionally comprising step of displaying said 3D image by means of amember of a group consisting of: a 3D screen, 3D glasses,autostereoscopy, holography, a volumetric display, integral imaging,wiggle stereoscopy, an active shutter 3d system, a polarized 3d system,an interference filter system, a color anaglyph systems a Chromadepthsystem, an over/under format, and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising step of detecting said tools by means of depthcues in said 3D image.

It is another object of the present invention to disclose the method,additionally comprising step of detecting at least one structureselected from a group consisting of an organ, a blood vessel, a nerve, alimb and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising step of emphasizing at least one said structureby means of a member of a group consisting of: virtual reality overlays,color markings, pattern markings, outlines, arrows, and any combinationthereof.

It is another object of the present invention to disclose the method,additionally comprising step of storing data from at least one otherimaging modality in a database.

It is another object of the present invention to disclose the method,additionally comprising steps of matching and aligning said data fromsaid at least one other imaging modality with said 3D image, and ofviewing substantially all information from said at least one otherimaging modality and said 3D image as a single 3D image.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said at least one otherimaging modality from a group consisting of: CT, MRI, ultrasound and PETand any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising steps of displaying additional information andof selecting said information from a group consisting of:recommendations useful in directing the progress of an operation,information useful in assisting an operator in making a decision,information useful for a record of the operation and any combinationthereof.

It is another object of the present invention to disclose the method,additionally comprising step of displaying said information visually oraudibly.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said information from a groupconsisting of: camera pose, body temperature, time, elapsed time sincestart of operation, patient notes, notes made during the operation, andother information accessible to the system and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising steps of providing at least one joystick unit incommunication with said maneuvering system, and operating saidmaneuvering system by means of said joystick unit.

It is another object of the present invention to disclose the method,additionally comprising step of operating said joystick, said operatingcausing motion of said endoscope by means of said maneuvering system.

It is another object of the present invention to disclose the method,wherein said joystick unit is wearable by a user of said system.

It is another object of the present invention to disclose the method,additionally comprising step of coupling said joystick unit to at leastone surgical tool used in said medical procedure.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said at least one surgicaltool to be said endoscope.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said movement of said joystickto be proportional to said movement of said endoscope.

It is another object of the present invention to disclose the method,additionally comprising step of setting said endoscope's speed to apredetermined value if said joystick unit's speed of motion is above apredetermined value.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said joystick unit to be aforce joystick.

It is another object of the present invention to disclose the method,additionally comprising step of comprising said joystick unit of a baseand lever coupled to said base, such that moving said lever results inmovement of said endoscope; said movement of said lever beingproportional to said movement of said endoscope.

It is another object of the present invention to disclose the method,additionally comprising steps of comprising said joystick unit of a baseand a button; and jointly connecting said button to said base, such thatmoving of said button results in movement of said endoscope; saidmovement of said button being proportional to said movement of saidendoscope.

It is another object of the present invention to disclose the method,additionally comprising step of comprising said joystick unit of atouchscreen, such that a touching and moving on said touchscreen resultsin movement of said endoscope; touch and movement on said touchscreenbeing proportional to said movement of said endoscope.

It is another object of the present invention to disclose the method,additionally comprising step of determining said location within saidsurgical environment of said human body from pressure on a portion ofsaid touchscreen.

It is another object of the present invention to disclose the method,wherein said portion of said touchscreen is that which displays an imageof said location.

It is another object of the present invention to disclose the method,additionally comprising means configured to restrain the velocity ofsaid endoscope, such that when said means are activated, the velocity ofsaid endoscope is restrained.

It is another object of the present invention to disclose the method,additionally comprising step of providing said joystick unit comprisingn sensors, where n is an integer larger than one.

It is another object of the present invention to disclose the method,additionally comprising step of activating at least one of said nsensors in case of power failure.

It is another object of the present invention to disclose the method,additionally comprising step of activating at least one of said nsensors when said system is connected to power.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said sensors from a groupconsisting of a motion sensor, a heat sensor, an electric sensor, asound sensor, a pressure sensor, an optical sensor and any combinationthereof.

It is another object of the present invention to disclose the method,additionally comprising step of characterizing said joystick unit by anexternal surface.

It is another object of the present invention to disclose the method,additionally comprising step of detecting motion upon said externalsurface by means of said at least one motion sensor.

It is another object of the present invention to disclose the method,additionally comprising step of detecting motion perpendicular to saidexternal surface by means of said at least one motion sensor.

It is another object of the present invention to disclose the method,additionally comprising step of sensing temperatures in the range ofabout 35 to about 42 degrees by means of said at least one heat sensor.

It is another object of the present invention to disclose the method,additionally comprising step of maneuvering of said endoscope at suchtimes as said at least one heat sensor senses temperatures in the rangeof about 35 to about 42 degrees.

It is another object of the present invention to disclose the method,additionally comprising step of coupling said at least one heat sensorto a processing unit comprising instructions configured, when executed,to provide said system user with a thermal image.

It is another object of the present invention to disclose the method,additionally comprising steps of maneuvering said endoscope at suchtimes as analysis of said image by said processing unit detects an imageof a human hand; and preventing maneuvering of said endoscope at suchtimes as said analysis of said image by said processing unit fails todetect said image of said human hand.

It is another object of the present invention to disclose the method,additionally comprising step of sensing power failure by means of saidat least one electric sensor.

It is another object of the present invention to disclose the method,additionally comprising step of sensing electric conductivity of a humanbody by means of said at least one electric sensor.

It is another object of the present invention to disclose the method,additionally comprising steps of maneuvering said endoscope at suchtimes as said sensor senses the conductivity of said subject's body; andpreventing maneuvering of said endoscope at such times as said sensorfails to sense the conductivity of said subject's body.

It is another object of the present invention to disclose the method,additionally comprising step of sensing at least one predetermined soundpattern by means of said at least one sound sensor.

It is another object of the present invention to disclose the method,additionally comprising step of maneuvering said endoscope according tosaid at least one predetermined sound pattern sensed by said at leastone sound sensor.

It is another object of the present invention to disclose the method,additionally comprising step of sensing pressure applied to saidjoystick unit by means of said at least one pressure sensor.

It is another object of the present invention to disclose the method,additionally comprising step of selecting response of said system formaneuvering an endoscope (SFME) to said pressure sensed by said at leastone pressure sensor from a group consisting of: activating said SFMEwhen said pressure sensed by said at least one pressure sensor is abovea predetermined value; de-activating said SFME when said pressure sensedby said at least one pressure sensor is above a predetermined value; andde-activating said SFME when said pressure sensed by said at least onepressure sensor is below a predetermined value.

It is another object of the present invention to disclose the method,additionally comprising step of sensing visual changes according to atleast one predetermined visual pattern by means of said at least oneoptical sensor.

It is another object of the present invention to disclose the method,additionally comprising step of maneuvering said endoscope according tosaid at least one predetermined visual pattern.

It is another object of the present invention to disclose the method,additionally comprising step of providing an interface system configuredto enable communication between said joystick unit and said maneuveringsystem.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said communication means froma group consisting of a wired communication means, a wirelesscommunication means and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising steps of providing at least one second joystickunit and configuring said second joystick unit to zoom said endoscope bymeans of said maneuvering system.

It is another object of the present invention to disclose the method,additionally comprising step of providing a single device comprisingsaid joystick unit and said second joystick unit.

It is another object of the present invention to disclose the method,additionally comprising step of wearing said second joystick unit bysaid system user.

It is another object of the present invention to disclose the method,additionally comprising step of coupling said second joystick unit to atleast one surgical tool.

It is another object of the present invention to disclose the method,additionally comprising step of providing said at least one surgicaltool to be said endoscope.

It is another object of the present invention to disclose the method,additionally comprising step of controlling and directing said endoscopeby means of said at least one joystick unit via said maneuvering system.

It is another object of the present invention to disclose the method,additionally comprising steps of selecting said at least one surgicaltool by activating said at least one joystick unit; activating said atleast one joystick unit by at least one of a group consisting of:depressing said joystick unit, voice activating the same, prolongeddepressing of the same, double clicking on the same and any combinationthereof.

It is another object of the present invention to disclose the method,additionally comprising step of positioning at least one proximitysensor on an outer circumference of said tool.

It is another object of the present invention to disclose the method,additionally comprising steps of providing a set of instructions which,when executed by data processing apparatus, and executing saidinstructions, thereby controlling said maneuvering of said endoscope.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said instructions from apredetermined set of rules selected from a group consisting of: mostused tool rule, right tool rule, left tool rule, field of view rule, nofly zone rule, a route rule, environmental rule, operator input rule,proximity rule; collision prevention rule, history-based rule,tool-dependent ALLOWED and RESTRICTED movements rule, preferred volumezone rule, preferred tool rule, movement detection rule, tagged toolrule, change of speed rule and any combination thereof.

It is another object of the present invention to disclose the method,wherein said route rule comprises steps of: providing a communicabledatabase; storing a predefined route in which said at least one surgicaltool is configured to move within said surgical environment; comprisingsaid predefined route of n 3D spatial positions of said at least onesurgical tool, n is an integer greater than or equal to 2; said ALLOWEDmovements are movements in which said at least one surgical tool islocated substantially in at least one of said n 3D spatial positions ofsaid predefined route, and said RESTRICTED movements are movements inwhich said location of said at least one surgical tool is substantiallydifferent from said n 3D spatial positions of said predefined route.

It is another object of the present invention to disclose the method,wherein said environmental rule comprises steps of: providing acommunicable database; receiving at least one real-time image of saidsurgical environment in said communicable database; performing real-timeimage processing of the same and determining the 3D spatial position ofhazards or obstacles in said surgical environment; determining saidALLOWED and RESTRICTED movements according to said hazards or obstaclesin said surgical environment, such that said RESTRICTED movements aremovements in which said at least one surgical tool is locatedsubstantially in at least one of said 3D spatial positions, and saidALLOWED movements are movements in which the location of said at leastone surgical tool is substantially different from said 3D spatialpositions.

It is another object of the present invention to disclose the method,additionally comprising steps of selecting said hazards or obstacles insaid surgical environment from a group consisting of tissue, a surgicaltool, an organ, an endoscope and any combination thereof.

It is another object of the present invention to disclose the method,wherein said operator input rule comprises steps of: providing acommunicable database; and receiving input from an operator of saidsystem regarding said ALLOWED and RESTRICTED movements of said at leastone surgical tool.

It is another object of the present invention to disclose the method,additionally comprising steps of: comprising said input of n 3D spatialpositions, n is an integer greater than or equal to 2; defining at leastone of said spatial positions as an ALLOWED location; defining at leastone of said spatial positions as a RESTRICTED location; such that saidALLOWED movements are movements in which said at least one surgical toolis located substantially in at least one of said n 3D spatial positions,and said RESTRICTED movements are movements in which the location ofsaid at least one surgical tool is substantially different from said n3D spatial positions.

It is another object of the present invention to disclose the method,additionally comprising steps of: comprising said input of at least onerule according to which ALLOWED and RESTRICTED movements of said atleast one surgical tool are determined, such that the spatial positionof said at least one surgical tool is controlled by said controlleraccording to said ALLOWED and RESTRICTED movements.

It is another object of the present invention to disclose the method,additionally comprising steps of selecting said predetermined set ofrules from a group consisting of: most used tool, right tool rule, lefttool rule, field of view rule, no fly zone rule, route rule,environmental rule, operator input rule, proximity rule, collisionprevention rule, preferred volume zone rule, preferred tool rule,movement detection rule, history-based rule, tool-dependent ALLOWED andRESTRICTED movements rule, and any combination thereof.

It is another object of the present invention to disclose the method,wherein said operator input rule comprises steps of: converting anALLOWED movement to a RESTRICTED movement and converting a RESTRICTEDmovement to an ALLOWED movement.

It is another object of the present invention to disclose the method,wherein said proximity rule comprises steps of: defining a predetermineddistance between at least two surgical tools; said ALLOWED movements aremovements which are within the range or out of the range of saidpredetermined distance, and said RESTRICTED movements are movementswhich are out of the range or within the range of said predetermineddistance.

It is another object of the present invention to disclose the method,wherein said proximity rule comprises steps of: defining a predeterminedangle between at least three surgical tools; said ALLOWED movements aremovements which are within the range or out of the range of saidpredetermined angle, and said RESTRICTED movements are movements whichare out of the range or within the range of said predetermined angle.

It is another object of the present invention to disclose the method,wherein said collision prevention rule comprises steps of: defining apredetermined distance between said at least one surgical tool and ananatomical element within said surgical environment; said ALLOWEDmovements are movements which are in a range that is larger than saidpredetermined distance, and said RESTRICTED movements are movementswhich is in a range that is smaller than said predetermined distance.

It is another object of the present invention to disclose the method,additionally comprising steps of selecting said anatomical element froma group consisting of tissue, organ, another surgical tool and anycombination thereof.

It is another object of the present invention to disclose the method,wherein said right tool rule comprises steps of: determining saidALLOWED movement of said endoscope according to the movement of thesurgical tool positioned to right of said endoscope; further whereinsaid left tool rule comprises steps of: determining said ALLOWEDmovement of said endoscope according to the movement of the surgicaltool positioned to left of said endoscope.

It is another object of the present invention to disclose the method,wherein said tagged tool rule comprises steps of: tagging at least onesurgical tool within said surgical environment and determining saidALLOWED movements of said endoscope to be movements that constantlytrack the movement of said tagged surgical tool.

It is another object of the present invention to disclose the method,wherein said field of view rule comprises steps of: providing acommunicable database comprising n 3D spatial positions; n is an integergreater than or equal to 2; generating a field of view from thecombination of all of said n 3D spatial positions; maintaining aconstant field of view by determining said ALLOWED movement of saidendoscope to be within said n 3D spatial positions, such that saidALLOWED movements are movements in which said endoscope is locatedsubstantially in at least one of said n 3D spatial positions, and saidRESTRICTED movements are movements in which the location of saidendoscope is substantially different from said n 3D spatial positions.

It is another object of the present invention to disclose the method,wherein said preferred volume zone rule comprises steps of: providing acommunicable database comprising n 3D spatial positions; n is an integergreater than or equal to 2; generating said preferred volume zone fromsaid n 3D spatial positions; determining said ALLOWED movement of saidendoscope to be within said n 3D spatial positions and said RESTRICTEDmovement of said endoscope to be outside said n 3D spatial positions,such that said ALLOWED movements are movements in which said endoscopeis located substantially in at least one of said n 3D spatial positions,and said RESTRICTED movements are movements in which the location ofsaid endoscope is substantially different from said n 3D spatialpositions.

It is another object of the present invention to disclose the method,wherein said preferred tool rule comprises steps of: providing acommunicable database, storing a preferred tool in said database;determining said ALLOWED movement of said endoscope so as to constantlytrack the movement of said preferred tool.

It is another object of the present invention to disclose the method,wherein said no fly zone rule comprises steps of: providing acommunicable database comprising n 3D spatial positions, n is an integergreater than or equal to 2; defining a predetermined volume within saidsurgical environment from said n 3D spatial positions; determining saidRESTRICTED movement to be said movement within said no fly zone;determining said ALLOWED movement to be said movement outside said nofly zone, such that said RESTRICTED movements are movements in whichsaid at least one of said surgical tool is located substantially in atleast one of said n 3D spatial positions, and said ALLOWED movements aremovements in which the location of said at least one endoscope issubstantially different from said n 3D spatial positions.

It is another object of the present invention to disclose the method,wherein said most used tool rule comprises steps of: providing acommunicable database; counting the amount of movement of each saidsurgical tool; constantly positioning said endoscope to track movementof the most moved surgical tool.

It is another object of the present invention to disclose the method,additionally comprising steps of providing a maneuvering subsystemcommunicable with said controller, spatially repositioning said at leastone surgical tool during a surgery according to said predetermined setof rules; and alerting the physician of said RESTRICTED movement of saidat least one surgical tool.

It is another object of the present invention to disclose the method,additionally comprising steps of selecting said alert from a groupconsisting of: audio signaling, voice signaling, light signaling,flashing signaling and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising steps of defining said ALLOWED movement as amovement permitted by said controller and defining said RESTRICTEDmovement as a movement denied by said controller.

It is another object of the present invention to disclose the method,wherein said history-based rule comprises steps of: providing acommunicable database storing each 3D spatial position of each saidsurgical tool, such that each movement of each surgical tool is stored;determining said ALLOWED and RESTRICTED movements according tohistorical movements of said at least one surgical tool, such that saidALLOWED movements are movements in which said at least one surgical toolis located substantially in at least one of said 3D spatial positions,and said RESTRICTED movements are movements in which the location ofsaid at least one surgical tool is substantially different from said 3Dspatial positions.

It is another object of the present invention to disclose the method,wherein said tool-dependent ALLOWED and RESTRICTED movements rulecomprises steps of: providing a communicable database; storingpredetermined characteristics of at least one said surgical tool;determining said ALLOWED and RESTRICTED movements according to saidpredetermined characteristics of said surgical tool; such that ALLOWEDmovements are movements of said endoscope which track said surgical toolhaving said predetermined characteristics.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said predeterminedcharacteristics of said surgical tool from a group consisting of:physical dimensions, structure, weight, sharpness, and any combinationthereof.

It is another object of the present invention to disclose the method,wherein said movement detection rule comprises steps of: providing acommunicable database comprising the real-time 3D spatial positions ofeach said surgical tool; detecting movement of said at least onesurgical tool when a change in said 3D spatial positions is received,such that said ALLOWED movements are movements in which said endoscopeis re-directed to focus on said moving surgical tool.

It is another object of the present invention to disclose the method,additionally comprising steps of providing a maneuvering subsystemcommunicable with said controller, spatially repositioning said at leastone surgical tool during a surgery according to said predetermined setof rules, such that if said movement of said at least one surgical toolis a RESTRICTED movement, said maneuvering subsystem prevents saidmovement.

It is another object of the present invention to disclose the method,additionally comprising step of providing, for each said instructingfunction, a weighting function, said weighting function, order ofexecution of said instructions selectable by said weighting function

It is another object of the present invention to disclose the method,additionally comprising steps of providing:

-   (a) at least one wearable operator comprising at least one wireless    transmitter, configured to transmit a signal once said at least one    wearable operator is activated; said at least one wearable operator    is either wire or wirelessly in communication with at least one    surgical instrument;-   (b) at least one wireless receiver; configured to receive said    signal sent by said transmitter;-   (c) at least one laparoscopy computerized system, in communication    with said wireless receiver, configured to provide a visual onscreen    depiction of said at least one instrument to be selected following    the activation of said at least one wearable operator; and,-   (d) at least one video screen; wherein said system is configured to    control and to direct said endoscope via said laparoscopy    computerized system and said maneuvering system on said instrument    to be selected following the activation of said at least one    wearable operator.

It is another object of the present invention to disclose the method,additionally comprising step of providing said communication betweensaid at least one of said wearable operators and said instrument bymeans of either wire or wirelessly coupling.

It is another object of the present invention to disclose the method,additionally comprising step of wearing said wearable operator by saidsurgeon on a predetermined body part.

It is another object of the present invention to disclose the method,additionally comprising step of selecting said predetermined body partfrom a group consisting of: the hand of said surgeon, at least one ofthe fingers of said surgeon, the thigh of said surgeon, the neck of saidsurgeon, at least one of the legs of said surgeon, the knee of saidsurgeon, the head of said surgeon and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising step of selecting shape of said wearableoperator from a group consisting of a ring, a bracelet and anycombination thereof.

It is another object of the present invention to disclose the method,additionally comprising step of coupling said wearable operator to apredetermined location on said instrument by means of an adaptor.

It is another object of the present invention to disclose the method,additionally comprising step of adjusting said wearable operator so asto fit said predetermined location of said different instruments, eachof which is characterized by a different size and shape.

It is another object of the present invention to disclose the method,additionally comprising steps of comprising said wearable operator of abody having at least two portions at least partially overlapping eachother; and configuring said two portions to grasp and hold either saidinstrument or said predetermined body part there-between, therebyobtaining a tight-fit coupling between said two portions and saidinstrument or said predetermined body part.

It is another object of the present invention to disclose the method,additionally comprising steps of coupling said wearable operator to saidinstrument; and rotating one of said two portions relative to the other,such that fine-tuned movement of said two body portions is obtained;thereby providing said tight-fit coupling between said two portions andsaid instrument or said predetermined body part.

It is another object of the present invention to disclose the method,additionally comprising steps of coupling said wearable operator to saidinstrument; and rotating said two portions relative to each other, suchthat fine-tuned movement of said two body portions is obtained; therebyproviding said tight-fit coupling between said two portions and saidinstrument or said predetermined body part.

It is another object of the present invention to disclose the method,additionally comprising step of comprising said wearable operator of (a)at least one flexible and stretchable strip; and (b) loop-closing meansconfigured to close a loop with said at least one flexible andstretchable strip; said at least one flexible and stretchable strip andsaid loop-closing means are provided so as to fit said wearable operatorto at least one selected from a group consisting of (a) saidpredetermined location of said different instruments; (b) saidpredetermined body part of said surgeon, each of which is characterizedby a different size and shape.

It is another object of the present invention to disclose the method,additionally comprising step of selecting material for said flexible andstretchable strip from a group consisting of silicone, rubber and anycombination thereof.

It is another object of the present invention to disclose the method,additionally comprising step of configuring said wireless transmitter tolocate the position of at least one of said instruments.

It is another object of the present invention to disclose the method,additionally comprising step of selecting of said at least oneinstrument from a group consisting of: activating said at least onewearable operator; depressing on a predetermined location in saidwearable operator, voice activating the same, prolonged depression onthe same, double clicking on the same and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising step of directing said endoscope by using imageinformation shown on said video screen without said help of assistants.

It is another object of the present invention to disclose the method,additionally comprising steps of comprising said conventionallaparoscopy computerized system of at least one surgical instrumentspatial location software, configured to locate the 3D spatial positionof said at least one instrument; comprising said conventionallaparoscopy computerized system of at least one automated assistantmaneuvering system; coupling said automated assistant maneuvering systemto said endoscope; activating said at least one wearable operator;selecting said at least one instrument; and directing said endoscope tosaid at least one instrument.

It is another object of the present invention to disclose the method,additionally comprising step of matching each transmitted signal fromsaid wearable operator and said wireless transmitter to at least one ofsaid instruments.

It is another object of the present invention to disclose the method,additionally comprising steps of providing said tool with articulations,the location of said articulations selected from a group consisting of:substantially at the tip of said tool, substantially along the body ofsaid tool, and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising steps of controlling articulation by means of amethod selected from a group consisting of hardware control, softwarecontrol and any combination thereof.

It is another object of the present invention to disclose the method,additionally comprising steps of providing a tool articulated at aregion selected from a group consisting of near the tip of said tool, onthe body of said tool, and any combination thereof.

BRIEF DESCRIPTION OF THE FIGURES

In order to better understand the invention and its implementation inpractice, a plurality of embodiments will now be described, by way ofnon-limiting example only, with reference to the accompanying drawings,wherein

FIG. 1 schematically illustrates a structured light system;

FIGS. 2A-2H illustrate binary coded patterns;

FIGS. 3A-3H illustrate Gray-coded patterns;

FIG. 4 shows a stripe pattern of varying intensity;

FIG. 5 shows the sensitivity of the reconstructed object point toperturbations for an embodiment of the system for an embodiment of thesystem;

FIGS. 6A-6D depict IH, IL, the MSB pattern and the LSB pattern for anembodiment of the system;

FIGS. 7A-7E show the raw image corresponding to one embodiment of astripe pattern, the normalized intensity image for that stripe pattern,the binary image for that stripe pattern, the fidelity image for thatstripe pattern and profiles of a vertical line from the normalizedintensity image for an embodiment of the system;

FIGS. 8A-8B show the decoded stripe code T of a scanned object and thefidelity image F for an embodiment of the system;

FIG. 9 depicts images of a human face reconstructed using an embodimentof the system;

FIGS. 10A-10C depict the reconstructed surface and a vertical profile ofthe Z-map for an embodiment of the system;

FIG. 11 illustrates an embodiment of a calibration object;

FIG. 12 depicts forward projections of the theoretically calculatedcalibration object fiducial points for an embodiment of the system;

FIGS. 13A-13D show profiles of a reconstructed calibration object for anembodiment of the system;

FIG. 14 schematically illustrates a laparoscopic system using structuredlight for generating 3D images;

FIG. 15 schematically illustrates an embodiment of the distal end of anendoscope;

FIG. 16 shows a plot of errors in pixel coordinates for a typicalcalibration configuration;

FIG. 17 shows a stem plot of errors for a typical calibrationconfiguration;

FIG. 18 shows an embodiment of a calibration object;

FIG. 19 shows a plot of pixel residuals;

FIG. 20 shows a plot of stripe residuals; and

FIG. 21 shows a plot of spatial intersection errors

FIG. 22 shows absorption of light by substances found in tissue; and

FIGS. 23-25 show absorption of light by tissue at different wavelengths

FIGS. 26A-26B and 27A-27I schematically illustrate joystick units;

FIG. 28 schematically illustrates a system for controlling the field ofview on an endoscope;

FIGS. 29A-29B present a means to control the articulation of anarticulating endoscope;

FIG. 30 illustrates the use of the endoscope articulation control;

FIGS. 31A-31D illustrate articulation of the endoscope;

FIGS. 32A-32D schematically illustrate operation of an embodiment of atracking system with collision avoidance system;

FIGS. 33A-33D schematically illustrate operation of an embodiment of atracking system with no fly zone rule/function;

FIGS. 34A-34D schematically illustrate operation of an embodiment of atracking system with preferred volume zone rule/function;

FIG. 35 schematically illustrates operation of an embodiment of theorgan detection function/rule;

FIG. 36 schematically illustrates operation of an embodiment of the tooldetection function/rule;

FIGS. 37A-37B schematically illustrate operation of an embodiment of themovement detection function/rule;

FIGS. 38A-38D schematically illustrate operation of an embodiment of theprediction function/rule;

FIG. 39 schematically illustrates operation of an embodiment of theright tool function/rule;

FIGS. 40A-40B schematically illustrate operation of an embodiment of thefield of view function/rule;

FIG. 41 schematically illustrates operation of an embodiment of thetagged tool function/rule;

FIGS. 42A-42C schematically illustrate operation of an embodiment of theproximity function/rule;

FIGS. 43A-43B schematically illustrate operation of an embodiment of theoperator input function/rule;

FIGS. 44A-44D schematically illustrate an embodiment of a trackingsystem with a constant field of view rule/function;

FIG. 45 schematically illustrates an embodiment of a tracking systemwith a change of speed rule/function;

FIGS. 46A-46B schematically illustrate movement of an articulated tool;and

FIG. 47 schematically illustrates movement of an articulated tool.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description is provided, alongside all chapters of thepresent invention, so as to enable any person skilled in the art to makeuse of said invention and sets forth the best modes contemplated by theinventor of carrying out this invention. Various modifications, however,will remain apparent to those skilled in the art, since the genericprinciples of the present invention have been defined specifically toprovide a means and method for providing an enhanced 3D laparoscopicimage adapted to be useful for gastroscopy and colonoscopy.

The term ‘camera’ hereinafter refers to an image-capture device, capableof creating a 2D image of an object. Examples of a camera include, butare not limited to, a CCD array and an electromagnetic system such as aTV camera.

The term ‘endoscope’ refers hereinafter to any means adapted for lookinginside the body for medical reasons. This may be any instrument used toexamine the interior of a hollow organ or cavity of the body. Theendoscope may also refer to any kind of a laparascope. It should benoted that the following description may refer to an endoscope as asurgical tool.

The term ‘endoscope distal end’ hereinafter refers to the end of theendoscope that is inside the patient.

The term ‘endoscope proximal end’ hereinafter refers to the end of thecamera outside the patient. The camera is attached to the endoscope'sproximal end.

The term ‘structured light’ hereinafter refers to a method of producing3D images using a single 2D camera. In the structured light method, theobject is illuminated by a set of rays of light, each ray illuminating aspot on the object from a known position and a known direction, and eachray emitted at a known time. For each known time, a 2D camera image iscreated from light reflected from the spots created from rays existingat that time. Initially, a known calibration object is illuminated. Fromthe known shape, size and position of the calibration object and fromthe locations in the camera images of the reflected light, mathematicalmatrices can be calculated. These matrices enable calculation of the 3Dlocation of the surface of an unknown object, when the unknown object isilluminated by the same set of rays as illuminated the calibrationobject.

The term ‘structured light pattern’ hereinafter refers to the set ofrays of light with a known spatial and temporal pattern used, asdescribed above, to illuminate an object.

The term ‘field of view’ (FOV) hereinafter refers to the scene visibleto the camera.

The terms ‘about’ and ‘approximately’ hereinafter refers to a range of±20% of the value.

The term ‘world’ hereinafter refers to the region of space in whichobjects are located.

The term ‘world coordinates’ hereinafter refers to a coordinate systemfixed in the world.

The location in space of a physical object to be imaged will bedescribed by the world coordinate system.

The term ‘projector’ hereinafter refers to the set of light rays used toilluminate an object.

The term ‘projector coordinates’ hereinafter refers to a coordinatesystem in which the projector is located and which describes therelation of the light rays to each other, both spatially and temporally.

The term ‘camera coordinate system’ hereinafter refers to a coordinatesystem fixed with respect to the camera.

The term ‘stripe id’ hereinafter refers to the temporal and spatiallocation and direction of a light ray.

The term ‘articulation’ refers hereinafter to any device which has morethan 1 degree of freedom. Thus, said tool can bend either in the tipthereof or any location in the body of the same.

The term ‘toggle’ refers hereinafter to switching between one taggedsurgical tool to another.

The term ‘region of interest’ refers hereinafter to any region withinthe body of a subject which may be of interest for the operator of thesystem of the present invention. The region of interest may be, forexample, an organ to be operated on, a RESTRICTED area which a surgicalinstrument should avoid approaching, a surgical instrument or any otherregion within the human body or body of another living animal.

The term ‘surgical environment’ refers hereinafter to any anatomicalpart within the body of a subject which may be in the surroundings of asurgical instrument. The environment may comprise: organs, body parts,walls of organs, arteries, veins, nerves, a region of interest, or anyother anatomical part of the human body.

The term ‘spatial position’ refers hereinafter to a predeterminedspatial location and/or orientation of an object (e.g., the spatiallocation of the endoscope, the angular orientation of the endoscope, andany combination thereof).

The term ‘prohibited area’ refers hereinafter to a predetermined area towhich a surgical tool (e.g., an endoscope) is prohibited to be spatiallypositioned in.

The term ‘preferred area’ refers hereinafter to predetermined area towhich a surgical tool (e.g., an endoscope) is allowed and/or preferredto be spatially positioned in.

The terms ‘tool’, ‘surgical tool’ and ‘surgical instrument’ referhereinafter to any instrument or device introducible into the humanbody. The term may refer to any portion of the tool. For example it canrefer to the tip of the same, the body of the same and any combinationthereof. It should be further pointed that the following description mayrefer to a surgical tool/instrument as an endoscope. Non-limitingexamples of tools include a retractor, a clamp, a swab, a needle, anendoscope, and any other medical tool or instrument.

The term ‘provide’ refers hereinafter to any process (visual, tactile,or auditory) by which an instrument, computer, controller, or any othermechanical or electronic device can report the results of a calculationor other operation to a human operator.

The term ‘automatic’ or ‘automatically’ refers to any process thatproceeds without the necessity of direct intervention or action on thepart of a human being.

The term ‘automated assistant’ refers hereinafter to any mechanicaldevice (including but not limited to a robotic device) that can maneuverand control the position of a surgical or endoscopic instrument, andthat can in addition be adapted to receive commands from a remotesource.

The term ‘ALLOWED movement’ refers hereinafter to any movement of asurgical tool which is permitted according to a predetermined set ofrules.

The term ‘RESTRICTED movement’ refers hereinafter to any movement of asurgical tool which is forbidden according to a predetermined set ofrules. For example, one rule, according to the present invention,provides a preferred volume zone rule which defines a favored zonewithin the surgical environment. Thus, according to the presentinvention an ALLOWED movement of a surgical tool or the endoscope is amovement which maintains the surgical tool within the favored zone; anda RESTRICTED movement of a surgical tool is a movement which extracts(or moves) the surgical tool outside the favored zone.

The term ‘time step’ refers hereinafter to the working time of thesystem. At each time step, the system receives data from sensors andcommands from operators and processes the data and commands and executesactions. The time step size is the elapsed time between time steps.

The term ‘proximity sensor’ hereinafter refers to a sensor able todetect the presence of nearby objects without physical contact.Proximity sensors are sometimes referred to as ‘force sensors’. Aproximity sensor often emits an electromagnetic field or a beam ofelectromagnetic radiation (infrared, for instance), and looks forchanges in the field or return signal. The object being sensed is oftenreferred to as the proximity sensor's target. Different proximity sensortargets demand different sensors. For example, a capacitivephotoelectric sensor might be suitable for a plastic target; aninductive proximity sensor always requires a metal target. Proximitysensors can be introduced into the body and used for detecting metalfragments during surgery. See, for example, Sakthivel, M., A newinductive proximity sensor as a guiding tool for removing metal shrapnelduring surgery, Instrumentation and Measurement Technology Conference(I2MTC), 2013 IEEE International, pp. 53-57. ISSN: 1091-5281, printISBN: 978-1-4673.

The terms ‘degrees of freedom’ (DOF) refer hereinafter to a set ofindependent displacements that specify completely the displaced positionof the endoscope or laparoscope.

The term ‘insertion point’ refers hereinafter to the point where theendoscope enters the human body.

The terms ‘joystick’ and ‘joystick unit’ refer hereinafter to a motionand position sensitive device enabled to control the motion of anotherdevice, with the motion and position information including, but notlimited to, the direction of motion (in 1, 2 or 3 dimensions) and thespeed of the motion and the changes in direction and speed as a functionof time. Joystick units may, for example, in a non-limiting manner, beshaped like a rod or lever; which is bent, twisted, depressed or slid,the direction of the bend, twist, depression or sliding relatable to thedirection of motion and the magnitude thereof relatable to the speed ofthe motion. Joystick units can comprise a button which is depressed,slid or rocked, wherein the direction of the depression, sliding orrocking is related to the direction of motion and the magnitude thereofis related to the speed of the motion. They can comprise a surface alongwhich a finger or fingers or a hand or an implement slides, wherein thedirection of the motion on the surface is related to the direction ofmotion and the speed of the motion along the surface is related to thespeed of motion of the controlled device.

All temperatures referred to herein are temperatures in degrees Celsius.

The following abbreviations are used throughout the disclosure:

DOF refers to degree(s) of freedom;

SFME refers to System For Maneuvering an Endoscope, a system forenabling an operator to maneuver the endoscope as disclosed hereinbelow.

Laparoscopic surgery, also called minimally invasive surgery (MIS), is amodern surgical technique in which operations in the abdomen areperformed through small incisions (usually 0.5-1.5 cm) as compared tolarger incisions needed in traditional surgical procedures. The keyelement in laparoscopic surgery is the use of a laparoscope, which is adevice adapted for viewing the scene within the body, at the distal endof the laparoscope. Either an imaging device is placed at the end of thelaparoscope, or a rod lens system or fiber optic bundle is used todirect this image to the proximal end of the laparoscope. Also attachedis a light source to illuminate the operative field, inserted through a5 mm or 10 mm cannula or trocar to view the operative field.

The abdomen is usually injected with carbon dioxide gas to create aworking and viewing space. The abdomen is essentially blown up like aballoon (insufflated), elevating the abdominal wall above the internalorgans like a dome. Within this space, various medical procedures can becarried out.

In many cases, the laparoscope cannot view the entire working spacewithin the body, so the laparoscope is repositioned to allow the surgeonto view regions of interest within the space. In some laparoscopicsystems, this requires the surgeon to instruct an assistant to manuallymove the laparoscope. In other systems, the surgeon himself instructsthe laparoscope to move, by a manual control system such as a button,joystick or slider attached to the surgeon or to a surgical tool, bycontact with a touchscreen, or by voice commands.

In all such systems, in directing and maneuvering the surgicalcontrolling system, the controller needs to avoid obstacles such as bodyorgans and tools or other surgical equipment in the body cavity. Itsspeed should be controlled so that, on the one hand, the speed is lowenough to make avoidance routine and to ensure that the instrumentaccurately reaches the desired location and, on the other hand, thespeed needs to be great enough that maneuvers are accomplished in areasonable time.

In order to avoid the obstacles, in a conventional system, the endoscopemust be routed around them, increasing the complexity of maneuvering andthe time taken for maneuvering.

In the present, system, the system comprises at least one articulatingsection, typically an articulating tool such as an articulatingendoscope. The articulating tool can have an articulating tip, where thearticulations are near the tip, it can have an articulating body, wherethe articulations are in the body or shaft of the tool, or both. Thearticulations allow bending in at least two degrees of freedom (DOF),preferably in four DOF, and possibly in all six DOF (bending in allthree directions and rotating in all three directions).

In comparison to a rigid tool, during maneuvering, an articulating toolcan use more direct routes, as the articulating section enables removalof the tip of an articulating tool from the region of an obstacle. Forexample, instead of routing an endoscope around a body organ, theendoscope can articulate such that its tip is withdrawn to a sufficientheight that the route of the endoscope can be directly across the organ.

Furthermore, the system has more flexibility in positioning. Forexample, the angle of the field of view can be changed by changing thearticulation of the endoscope, with only minimal change of the positionof the main part of the endoscope.

A device of the present invention, with an articulating endoscope andproviding a 3D image, is useful for colonoscopy or gastroscopy,treatments where the endoscope moves within a relatively restricted,non-straight space. By traversing the interior of a portion of thealimentary canal, abnormalities such as, but not limited to, polyps orulcers can be identified and a warning provided to a user of the systemand to the patient. In preferred embodiments, the system additionallycomprises recognition software that can identify abnormalities and canlabel the abnormality in real time on an image of the field of view. Thelabel can be by any means known in the art such as, but not limited to,text, a colored patch, a textured patch, an arrow or other marker, andany combination thereof.

Because of the flexibility provided by the articulating endoscope andbecause of the increased locatability possible with a 3D image, thedevice of the present invention, compared to the prior art, can moreeasily follow the contours of body organs and can more effectively viewthe sides and backs of organs. It can therefore find cysts, lumps orother abnormal masses in the abdominal cavity more effectively than theprior art. It can also efficiently map the interior of the abdominalcavity and can be used for cartography of the intestines, including thebowels and colon.

In many cases, the surgeon wants a close-up view of his working area; inother cases an overview is desirable, and a rapid transition fromclose-up to overview and vice-versa can also be desirable.

The device disclosed herein uses a standard laparoscopic camera and acomputer-controllable laparoscopic light source in conjunction withsoftware using the structured light method in order to provide alaparoscopic system which presents the user with 3D images from a singlecamera.

A high resolution camera can provide sufficient detail to create adetailed 3D image.

In some embodiments, a wide-angle lens such as, but not limited to, afish-eye lens, an omnidirectional lens and any combination thereofprovides an image of a large portion, if not all, of the working area,such as the interior of the abdomen. The image provided by a wide-anglelens is frequently distorted; in preferred embodiments of systems with awide-angle lens, software is used to correct the distortion.

The advantages of a 3D image include:

-   -   It increases the accuracy and reliability of tool detection and        tracking from the laparoscope image, thereby minimizing or        eliminating the need for auxiliary equipment for the purpose.    -   It increases the accuracy of navigation within the body cavity,        thereby minimizing the risk of inadvertent tissue damage.    -   It provides a three-dimensional display for the surgeon, thereby        increasing the accuracy with which he can work.

There are many methods of providing a 3D image. These include, but arenot limited to:

-   -   Use of two cameras to provide a stereo image. This has the        disadvantage that two separated sets of camera lenses are        needed, and therefore a laparoscope adapted to accept two        separated camera lenses.    -   Use of the motion of the camera to get more than one view of the        scene, each view from a different perspective, and then        combining the views via internal algorithms. This has the        disadvantage of requiring motion of the camera.    -   Use of the Structured Light technique. This requires calibration        of the system before use, and can be sensitive to spurious        pixels in the image, but has the advantage of requiring neither        camera motion nor a special laparoscope.

For the laparoscope disclosed herein, structured light is the preferredmethod of providing a 3D image.

Advantages of a structured light system include:

-   -   The accuracy of the three-dimensional reconstruction is very        high.    -   Recovery from bit errors is possible.    -   It is possible to provide 3D images of bodies of uniform shade,        which is not possible in a stereo system using two cameras.    -   Adding a Distance Index from one object to the other (e.g.,        tissue).    -   Measurement of distances between any two points in the image is        simple, since the distance is a geodesic distance over the        surface or the Euclidean distance.    -   It is possible to embed texture in the image, improving the ease        of identification of tissues.    -   It is possible to use a standard camera and a standard        laparoscope, lighting the object via a standard laparoscope        lighting system.

In the structured light technique, schematically illustrated (100) inFIG. 1 , a controlled light pattern or a series of controlled lightpatterns is emitted by a light source (110) and the 3D image isconstructed from the light pattern received by a sensor (120) afterreflection from an object (130) or objects illuminated by the light andin the field of view of the sensor.

In the simplest structured light camera, a “projector”, a source oflight, projects a spot of light, for example, a laser beam, onto anobject. The location of the projector and the direction of projectionare known. The location on the camera image of the reflected spot can beeasily located and identified. Since the point location on both theprojector and the camera is known, reconstruction of the coordinates inspace of the reflecting point on the object (the “shape worldcoordinates”) can be carried out by simple triangulation.

A more efficient scheme replaces the point laser beam by a line beamthat “sweeps” the object. Still, sweeping the entire scene by a laserbeam is a time consuming operation and therefore is limited to staticobjects.

For a better scan time, several stripe illumination-based techniqueshave been proposed. Single pattern approaches, using spatial encoding ofprojection planes or rays can be used. Although time-efficient, spatialencoding generally produces a sparse depth map with mediocre spatialresolution.

High reliability identification of projection planes withnon-restrictive assumptions on scene contents and illuminationconditions can be achieved by temporal encoding, which is based onprojection of a sequence of light patterns (“stripes”) onto the object.A robust, commonly used structured light system based on binary lightpatterns encoded with Gray code is used in preferred embodiments of thedevice disclosed herein. Color modulation, which allows a reduction inthe number of stripes projected onto the object, is also possible, witha color camera. Color modulation, as discussed hereinbelow, can also beused be used in addition to spatial and temporal encoding in order todetermine the nature of the tissues being imaged,

FIG. 2A-H shows binary coded patterns, while FIG. 3A-H shows Gray-codedpatterns. In both figures, ‘HSB’ is the highest significant bit, and‘LSB’ is the least significant bit. The patterns differ in that, in theGray-coded patterns, the edge-most light stripe is split, appearing atboth edges, so that the pattern is symmetrical about its center. Havinga (half) light stripe at each edge improves the robustness of thesystem.

An important issue in structured light systems is calibration.Reconstruction is possible only when the camera projection matrix andthe projection planes are known in the world coordinate system.Calibration of structured light scanners is usually more complicatedthan that of a passive stereo pair. A standard approach consists ofthree steps: estimating the camera intrinsic matrix (cameracalibration), estimating the plane equations for each of the projectionplanes (projector calibration) and finally, estimating the Euclideantransformation between the camera and the projector (projector-cameracalibration).

Other examples of structured light systems use stripes that vary inintensity across the stripe, with each stripe being black at one edgeand white at the other. In embodiments with varying intensity, theintensity changes in time as the plane of light is swept across theworking space which includes the object. FIG. 4A-B shows the intensityof the light as a function of time t, with t₁ the period of the stripepattern of FIG. 4A and t₂ the period of the stripe pattern of FIG. 4B.

Example 1

An alternative approach, however, provides the preferred embodiment forthe present invention. This alternative approach is based on estimationof the world to camera image and world-to-projector coordinate systemtransformation, extended to simultaneously estimating a backprojectionoperator, an operator that enables determination of the location inspace of the surface of an object, from the locations on camera imagesof spots of light reflected from the object, where the spots of lightare generated by rays emitted from known positions, in known directionsand at known times.

The following sections describe a method of determining thebackprojection operator. In the method described hereinbelow, acalibration object of known shape and size (the world object) isilluminated by a set of rays of light (the projector, in the projectorcoordinate system) emitted from known positions, in known directions andat known times. These rays are reflected from the calibration object(the object, in the world coordinate system) and generate at least one2D camera image (the camera image, in the camera coordinate system).From the known ray positions, the known spot positions on the cameraimages and the known shape and size of the calibration object, operatorsare determined that enable determination of the 3D position in space ofan unknown object.

In the derivation hereinbelow, the model will be described, then amethod of generating a 3D image of an object (reconstruction of theobject) from the camera images and the stability of the reconstructionwill be described. This is followed by a method of generating thereconstruction operators from a calibration object, and animplementation of an embodiment of the method.

Model of a Structured Light System

A typical structured light system consists of a camera and a projector.The role of the projector is to light the scanned object in such a waythat, from the image (or sequence of images) acquired by the camera, astripe code can be extracted. The encoding can be done either spatiallyusing a single pattern or temporally using a series of varying patterns.The raw output of a structured light scanner is a stripe code assignedfor every pixel in the image. Intersection of a ray in world coordinatesystem (WCS) with a plane in WCS yields the world coordinates of anobject point. Using this triangulation method, the raw sensor data isconverted into 3D data in the WCS.

For simplicity, in the derivations below, it will be assumed that boththe camera and the projector obey the pin-hole optical model (non-lineardistortion correction can be applied for lenses that do not obey thismodel). The transformation from 3D world coordinates to camera imageplane coordinates will be described by a 3×4 perspective projectionmatrix (PPM).

The projector will be modeled by a 2×4 PPM, mapping world coordinates tostripe identification code (id).

Three coordinate systems are defined: a homogenous world coordinatesystem X_(w) in which the object's position is specified; a homogenouscamera coordinate system X_(c), in which pixel locations in the imageplane are specified, and a homogenous projector coordinate system X_(p),in which stripe ids are specified. The latter is particular, since itcontains only one independent coordinate.

The transformation from world coordinates to camera coordinates is givenby

X _(c) =C _(c) X _(w),  (1)

where C_(c) is the camera PPM and is of the form

$\begin{matrix}{C_{c} = {{{\alpha\begin{bmatrix}f_{x} & {kf}_{y} & x_{c}^{0} \\0 & f_{y} & y_{c}^{0} \\0 & 0 & 1\end{bmatrix}}\begin{bmatrix}R_{c} & t_{c}\end{bmatrix}}.}} & (2)\end{matrix}$

The rotation matrix R_(c) and the translation vector t_(c) define thetransformation between WCS X_(w) and the camera-centric reference frameX_(c). The parameters f_(x) and f_(y) are the camera focal length scaledto each of the CCD dimensions, and x_(c) ⁰ and y_(c) ⁰ are the origin ofX_(c) in image coordinates. The parameter α is a proportion coefficientand k is the shear of the camera coordinate system.

Similarly, the transformation form world coordinates to projectorcoordinates is given by

X _(p) =C _(p) X _(w),  (3)

where C_(p) is the projector PPM of the form

$\begin{matrix}{C_{p} = {{\alpha\begin{bmatrix}f_{p} & 0 & x_{p}^{0} \\0 & 0 & 1\end{bmatrix}}\begin{bmatrix}R_{p} & t_{p}\end{bmatrix}}} & (4)\end{matrix}$

R_(p) and t_(p) define the transformation between WCS and X_(p). Theparameter f_(p) is the projector focal length scaled to the projector'sdimensions, and x_(p) ⁰ is the origin of X_(p) in projector coordinates,which physically is the x-coordinate of the intersection of the opticalaxis and the projector.

Here we implicitly assume that the stripe code varies along thehorizontal direction of the projector. McIvor and Valkenburg [A. M.McIvor, R. J. Valkenburg, I Robert, J. Valkenburg, J. Valkenburg,Calibrating a structured light system, Image & Vision Computing, NewZealand, 1995] show that C_(p) is a valid camera PPM if and only if thesubmatrix formed by its first three columns has full rank. Similarly,P_(p) is a valid projector PPM if and only if the submatrix formed byits first three columns is of rank 2.

Equations 1 and 3 define the transformation

T:X _(w)→(X _(c) ,X _(p)),  (5)

which maps an object point in WCS into pixel location in the cameraimage plane and a stripe id (coordinate in the projector system ofcoordinates). We refer to this transformation as forward projection.

The world coordinates of the object point are usually unknown and haveto be determined, whereas the pair (x_(c), x_(p)) is what the structuredlight sensor measures and can be extracted from the raw data. Therefore,given the camera and the projector PPMs and a pair of measurements(x_(c), x_(p)), one can attempt inverting eq. (5) in order to calculatex_(w). We will term the inverse transformation

T ⁻¹:(X _(c) ,X _(p))→X _(w),  (6)

as backprojection and the process of determining world coordinates frommeasured data as reconstruction.

Reconstruction requires the knowledge of C_(c) and C_(p). Therefore,calibration must be performed before, during which the forwardprojection operator is estimated. This is done by measuring a set ofpairs {(x_(c)x_(p))_(n)}_(n=1) ^(N) corresponding to a set of pointswith known world coordinates {(x_(w))_(n)}_(n=1) ^(N). Physically, acalibration object with a set of fiducial points, whose location isknown, is scanned. WCS is then chosen to be some local coordinate systemof the calibration object, in which the coordinates of each fiducialpoint are specified.

Reconstruction

In this section we assume that the forward projection operator T isknown (i.e. the projective matrices C_(c) and C_(p) are given). Thereconstruction problem can be stated as follows: given measured (x_(c),x_(p)), calculate x_(w) according to

x _(w) =T ⁻¹(x _(c) ,x _(p)).  (7)

Explicitly, x_(w) has to satisfy the linear system of equations

x _(c) =C _(c) x _(w)  (8)

x _(p) =C _(p) x _(w).  (9)

However, since all vectors are given in homogenous coordinates, it ispossible that no x_(w) satisfies eqs. (8) and (9) simultaneously. Let usdenote x_(c)=[w_(c)x_(c), w_(c)y_(c), w_(c)]^(T) and x_(p)=[w_(p)x_(p),w_(p)]^(T) and let ck, pk be the k-th row of C_(c) and C_(p),respectively. Then, the linear system of equations can be rewritten as

w _(c) x _(c) =c ₁ x _(w)

w _(c) y _(c) =c ₂ x _(w)

w _(c) =c ₃ x _(w)  (10)

and

w _(p) x _(p) =p ₁ x _(w)

w _(p) =p ₂ x _(w).  (11)

Substituting w_(c) into eq. (10) and wp into eq. (11) yields

x _(c) c ₃ x _(w) =c ₁ x _(w)

y _(c) c ₃ x _(w) =c ₂ x _(w)

x _(p) p ₂ x _(w) =p ₁ x _(w),  (12)

which can be written in matrix notation as Qx_(w)=0, where

$\begin{matrix}{Q = \begin{bmatrix}{{x_{c}c_{3}} - c_{1}} \\{{y_{c}c_{3}} - c_{2}} \\{{x_{p}p_{2}} - p_{1}}\end{bmatrix}} & (13)\end{matrix}$

The matrix Q can be split into a 3×3 matrix R and a 3×1 vector s: Q=[R,s]. Substituting x_(w)=[w_(w)x_(w), w_(w)y_(w), w_(w)z_(w), w_(w)]^(T)yields

$\begin{matrix}{{\left\lbrack {R,s} \right\rbrack\begin{bmatrix}{w_{w}x_{w}} \\{w_{w}x_{y}} \\{w_{w}z_{w}} \\w_{w}\end{bmatrix}} = {{{R\begin{bmatrix}{w_{w}x_{w}} \\{w_{w}x_{y}} \\{w_{w}z_{w}}\end{bmatrix}} + {w_{w}s}} = 0.}} & (14)\end{matrix}$

Therefore, the object point in non-homogenous world coordinatesx_(w)=[x_(w), y_(w), z_(w)]^(T) is a solution of the linear system

Rx _(w) =−s.  (15)

Backprojection is therefore given by

x _(w) =−R ⁻¹ s  (16)

It must be remembered that both R and s are functions of x_(c), y_(c),and x_(p).

If C_(c) and C_(p) are valid camera and projector PPMs, R is invertibleexcept of cases where the ray originating from the camera focal point tothe object point is parallel to the plane originating at the projectorfocal point and passing through the object point. The latter case ispossible either when the object point is located at infinity, or whenthe camera and the projector optical axes are parallel (this happenswhen R_(c)=R_(p)). This gives a constraint on camera and projectormutual location: in order to make triangulation possible, the camerashould not have its optical axis parallel to that of the projector.

Reconstruction Stability

As discussed hereinabove, the matrix R in Eq. (15) becomes singular whenthe ray in the camera coordinate system and the plane in the projectorcoordinates system are parallel. A reasonable question that may arise ishow stable is the solution under random perturbations of x_(c) andx_(p). In this work, we will address only perturbations in x_(p), sincethey are the most problematic ones in structured light systems.

For simplicity, let us assume that WCS coincides with the cameracoordinate system and the transformation to the projector coordinatesystem is given by

x _(p) =R _(p) +t _(p).  (17)

Without loss of generality, it can be assumed that the centers of thecamera coordinate system and the projector coordinate system coincidewith their optical axes, i.e. x_(c) ⁰=y_(c) ⁰=x_(p) ⁰=0.

Without loss of generality, it can be assumed that the object point isfound on some ray in x_(c)=av_(c); the ray is uniquely defined by thecamera image plane coordinates x_(c) and the point location is uniquelydefined by the parameter α. The stripe id corresponding to the givenobject point can be denoted by x_(p). Then, the following system oflinear equations

n ^(T) x _(p)=0

n ^(T)(R _(p) x _(c) +t _(p))=0,  (18)

must hold simultaneously; n denotes the normal to the plane defined bythe stripe id x_(p). Substituting x_(c)=av_(c) yields

n ^(T) x _(p) =n ^(T)(αR _(p) v _(c) +t _(p)),  (19)

hence

$\begin{matrix}{\alpha = \frac{n^{T}x_{\rho}}{n^{T}R_{p}v_{c}}} & (20)\end{matrix}$

However, in practice, the stripe id x_(p) is estimated using structuredlight, and therefore it is especially sensitive to noise. Let us assumethat instead of the real stripe id x_(p), a perturbed stripe id {tildeover (x)}_(p)=x_(p)+δx_(p) was measured. This, in turn, means that{tilde over (x)}_(p)=x_(p)+[δx_(p), 0, f_(p)]^(T), which yields

$\begin{matrix}{\overset{\sim}{\alpha} = {\frac{n^{T}{\overset{\sim}{x}}_{\rho}}{n^{T}R_{p}v_{c}}.}} & (21)\end{matrix}$

Hence, the perturbation in x_(p) causes a perturbation in the locationof the object point along the ray x_(c)=αv_(c) by

$\begin{matrix}{{{\delta\alpha} = \frac{n_{1}\delta x_{p}}{{n}_{2}{v}_{2}\sin\Theta_{nv}}},} & (22)\end{matrix}$

where Θ_(nv) is the angle between the plane defined by the normal n andthe ray defined by the direction v_(c). Therefore,

$\begin{matrix}{{\delta{x_{w}}_{k}} = {{{❘{\delta\alpha}❘}{v_{c}}_{k}} = {{❘\frac{n_{1}}{{n}_{2}\sin\Theta_{nv}}❘}{{❘{\delta x_{p}}❘}.}}}} & (23)\end{matrix}$

The ratio

${\cos\theta_{p}} = \frac{n_{1}}{{n}_{2}}$

has a geometrical interpretation of cosine of the projection angle;substituting it into Eq. (23) yields the sensitivity of thereconstructed object point to perturbations in the stripe id:

$\begin{matrix}{\frac{\delta{x_{w}}_{k}}{❘{\delta x_{p}}❘} = {{❘\frac{\cos\theta_{p}}{\sin\Theta_{nv}}❘}.}} & (24)\end{matrix}$

Calibration

In this section we assume that the forward projection operator T isunknown and has to be estimated from a given set of measured{(x_(c),x_(p))_(n)}_(n=1) ^(N) and corresponding known {x_(w)}_(n=1)^(N). Explicitly, it is desired to find such C_(c) and C_(p) that obey

(x _(c))_(k) =C _(c)(x _(w))_(k)  (25)

(x _(p))_(k) =C _(p)(x _(w))_(k),  (26)

for k=1, . . . , N. Since data measurement is not perfect (e.g., boththe camera and the projector resolution is finite), no projectionoperator will fit the data perfectly. Therefore, it is necessary to findsuch a T⁻¹ that will relate the measured and the known data in anoptimal way. It is thus important to address the optimality criterion.

McIvor and Valkenburg (see above) study the possibility of optimizingseparately the camera and the projector forward projections in the senseof the L₂ norm. Mathematically, this can be formulated as

$\begin{matrix}{{{C_{c} = {{argmin}{\sum\limits_{k = 1}^{N}{{{{C_{c}\left( x_{w} \right)}_{k} - \left( x_{c} \right)_{k}}}_{2}^{2}{s.t}}}}},{C_{c} \in {PPM}}}{{C_{p} = {{argmin}{\sum\limits_{k = 1}^{N}{{{{C_{p}\left( x_{w} \right)}_{k} - \left( x_{p} \right)_{k}}}_{2}^{2}{s.t}}}}},{C_{p} \in {{PPM}.}}}} & (27)\end{matrix}$

Let us define

$\begin{matrix}{{B_{k} = \begin{bmatrix}\left( x_{w} \right)_{k} & 0 \\0 & \left( x_{w} \right)_{k} \\{{- \left( x_{c} \right)_{k}}\left( x_{w} \right)_{k}} & {{- \left( y_{c} \right)_{k}}\left( x_{w} \right)_{k}}\end{bmatrix}^{T}}{{I = \left\lbrack {c_{1},c_{2},c_{3}} \right\rbrack^{T}},}} & (28)\end{matrix}$

where c_(k) is the k-th row of C_(c). Using this notation, the set of Nequations (25) can be rewritten as

B _(k) l=0,  (29)

for k=1, . . . , N, which in turn can be expressed as a singlehomogenous linear equation

Al=0,  (30)

Where A=[B₁ ^(T), . . . , B_(N) ^(T)]^(T). The vector of variables l isthe camera projection matrix C_(c) needed to be determined. Since thecamera PPM is defined up to a scaling factor, we will demand ∥l∥₂=1 inorder to avoid the trivial solution. With physically measured data, thematrix A will usually have full rank and therefore, no l will be anexact solution of eq. (30). However, one can find the best least-squaressolution by solving

l=argmin∥Al|∥ ₂ ^(Z)s.t.∥l∥ ₂=1  (31)

and ensuring that the obtained C_(c) is a valid PPM. Solving eq. (31) isequivalent to solving eq. (27) for the camera matrix, and its solutionminimizes the square error between the measured image plane coordinatesof the set of fiducial points and those obtained by projecting the setof the corresponding points in WCS onto the camera image plane.

Similarly, replacing B_(k) and l in eq. (28) with

$\begin{matrix}{{A = \left\lbrack {B_{1}^{T},\ldots,B_{N}^{T}} \right\rbrack^{T}}{B_{k} = \begin{bmatrix}\left( x_{w} \right)_{k} \\{{- \left( x_{p} \right)_{k}}\left( x_{w} \right)_{k}}\end{bmatrix}^{T}}{l = \left\lbrack {p_{1},p_{2}} \right\rbrack^{T}}} & (32)\end{matrix}$

yields the L₂ minimization problem, eq. (27), for the projector matrix.

The optimization problem of eq. (31) is a minimum eigenvalue problem andit can be shown that l minimizing ∥Al∥₂ is the eigenvector correspondingto the minimum eigenvalue of A^(T)A. It must be noted, however, thatsince usually the minimum eigenvalue of A^(T)A is very small, numericalinaccuracies are liable to rise.

Solution to the problem 27 finds two PPMs that minimize the squarederror between the measured data and the forward projection of the knownfiducial points in WCS into the camera and the plane coordinate systems.However, what is actually needed is to minimize the squared errorbetween the known fiducial points in WCS and the backward-projectedmeasurements. Mathematically, this can be formulated as

$\begin{matrix}{T = {\arg\min{\sum\limits_{k = 1}^{N}{{{\mathcal{T}^{- 1}\left( {x_{c},x_{p}} \right)}_{k} - \left( x_{w} \right)_{k}}}_{2}^{2}}}} & (33)\end{matrix}$ s.t.C_(c), C_(p) ∈ textPPM.

The above problem is no more separable and is non-convex; therefore, ithas to be solved by numerical global optimization methods. Still, anefficient solution in a few iterations is possible using the Newtonmethod, since the number of variables in the problem is small(3×4+2×4=20) and both the cost function, its gradient, and the Hessiancan be computed analytically. As the starting point for iterativeoptimization, a solution of eq. (27) can be used.

Since the calibration process is performed only once, additionalcomputational complexity can be used in order to obtain betterprojection estimation and better reconstruction results.

FIG. 5 shows the sensitivity of the reconstructed object point toperturbations in x_(p); the smaller the angle θ_(p), the larger theerror δ∥x_(w)∥₂.

Implementation of Reconstruction

In our implementation, we used temporal encoding, which allowed toobtain a dense z-map of the scanned objects. Eight binary patterns,encoding the stripe id using the Gray code were projected onto theobject and 8 corresponding images I_(k) were acquired corresponding tothe most significant bit). In addition, we acquired a full-darknessimage I_(L) (the projector's LCD was blackened) and a full-illuminationimage IH (the projector's LCD was set to maximum intensity). These twoimages served for compensation of ambient light and the non-constantalbedo of the object.

The quantity I_(L)(x,y) reflects the illumination of the object in pixel(x,y) at darkness and differs from zero only due to presence of ambientillumination. Since the reflectance of objects at illumination levelsused in normal conditions obeys linear superposition law, subtractingI_(L) from the rest of the images compensates for the ambience light.The quantity I_(H)(x,y)−I_(L)(x,y) is proportional to the object albedoat the pixel (x,y).

We define a set of 8 normalized intensity images

$\begin{matrix}{{J_{k}\left( {x,y} \right)} = {\frac{{I_{k}\left( {x,y} \right)} - {I_{L}\left( {x,y} \right)}}{{I_{H}\left( {x,y} \right)} - {I_{L}\left( {x,y} \right)}}.}} & (34)\end{matrix}$

A normalized intensity image J_(k)(x,y) has the values in the range [0,1] and reflects the amount of light irradiated onto the object surfaceat pixel (x,y). The value of 1 stands for full illumination, whereas thevalue of 0 stands for no illumination. Theoretically, J_(k) should bebinary images: 1 where a light stripe is present and 0 in places wherethere is a dark stripe. In practice, however, J_(k) are not binary andwe therefore define

$\begin{matrix}{{B_{k}\left( {x,y} \right)} = \left\{ {\begin{matrix}{{1:{J_{k}\left( {x,y} \right)}} > 0.5} \\{{0:{J_{k}\left( {x,y} \right)}} \leq 0.5}\end{matrix}.} \right.} & (35)\end{matrix}$

Better results were obtained when J_(k)(x,y) was smoothed with aGaussian filter prior to binarization. FIG. 6A-D depicts I_(H) (FIG. 6A)and I_(L) (FIG. 6B) as well as the LSB (FIG. 6D) pattern and the MSB(FIG. 6C) pattern.

FIG. 7A-E presents the normalized intensity image J₃(x,y), thecorresponding binary image B₃(x,y) and a profile of a vertical line fromthese two images. FIG. 7A shows the raw image corresponding to oneembodiment of a stripe pattern, FIG. 7B shows the normalized intensityimage for that stripe pattern, FIG. 7C shows the binary image for thatstripe pattern and FIG. 7D shows the fidelity image for that stripepattern. All images are normalized. FIG. 7E shows the profile of avertical line from the normalized intensity image (FIG. 7B) (dashedline) and the binary image (solid vertical lines).

For every pixel (x,y), we define the stripe code as the Gray codesequence

S(x,y)=[B ₁(x,y), . . . ,B ₈(x,y)].  (36)

Decoding S(x,y) yields a number T(x,y) E [0, 1], which will be referredto as stripe id. Note that T(x,y) is not really continuous but ratherhas the values T(x,y)∈{2^(−N)n:n=0, . . . , 2^(N)−1} (in the example,N=8).

For every pixel, x_(p)(x,y)=T(x,y) defines the projector coordinate ofan unknown object point corresponding to the pixel (x,y), transformed bythe projector PPM. Similarly, the pixel indices (x,y) define the cameraimage plane coordinates x_(c)=x, y_(c)=y of the object point projectedonto the camera coordinate system. Given the camera and the projectorPPMs, world coordinates of the object point can be calculated accordingto eq. (21).

Pixel Fidelity Estimation

It is obvious that although both J_(k)(x,y)=0.95 and J_(k)(x,y)=0.55will be binarized as B_(k)(x,y)=1, they should definitely be treateddifferently. In the first case, one may say that the pixel (x,y) inimage k is indeed illuminated with high probability, whereas in thesecond case the probability of that pixel to be non-illuminated isalmost equal to the probability of it being illuminated.

In order to give a quantitative measure of the pixel fidelity, let usassume that the measured normalized intensity image J_(k)(x,y) isobtained from some ideal binary image B_(k) ⁰(x,y) contaminated byzero-mean Gaussian noise ξ(x,y) with variance σ². In caseJ_(k)(x,y)>0.5, the pixel fidelity can be defined as

${{F_{k}\left( {x,y} \right)} = {{P\left\{ {{{J_{k}\left( {x,y} \right)} + {\xi\left( {x,y} \right)}} > 0.5} \right\}} = {\Phi\left( \frac{0.5 - {J_{k}\left( {x,y} \right)}}{\sigma} \right)}}},$

where Φ denotes the c.d.f. of the normal distribution. The lower valueof F_(k) is 0.5 and it is obtained when J_(k)(x,y)=0.5. Similarly, forJ_(k)(x,y)<0.5, the fidelity is

$\begin{matrix}{{F_{k}\left( {x,y} \right)} = {P\left\{ {{{J_{k}\left( {x,y} \right)} + {\xi\left( {x,y} \right)}} < 0.5} \right\}}} \\{= {{1 - {\Phi\left( \frac{0.5 - {J_{k}\left( {x,y} \right)}}{\sigma} \right)}} = {{\Phi\left( \frac{{J_{k}\left( {x,y} \right)} - 0.5}{\sigma} \right)}.}}}\end{matrix}$

Binarization errors in images corresponding to the most significant bitof the stripe code affect more the resulting stripe id T than errors inthe least significant bit. Therefore, pixel fidelity in each stripeshould be weighted by the stripe significance. We define the pixelfidelity as the weighted sum of the pixel fidelities in all stripes

${{F\left( {x,y} \right)} = {{\sum\limits_{k = 1}^{N}{w_{k}{F_{k}\left( {x,y} \right)}}} = {\sum\limits_{k = 1}^{N}{w_{k}{\Phi\left( {❘\frac{0.5 - {J_{k}\left( {x,y} \right)}}{\sigma}❘} \right)}}}}},$

where w_(k)=2^(−k) is the significance of stripe k, and N is 8 in ourimplementation. The variance σ² was set empirically to 1. FIG. 8A-Bshows the decoded stripe code T (FIG. 8A) of a scanned object and thefidelity image F (FIG. 8B).

Pixel fidelity can also provide important information and can be used,for example, for weighted mesh smoothing or decimation. For example, thepixel fidelity map can be used to obtain sub-stripe resolution, as shownbelow.

Sub-Stripe Resolution

One of important concerns in structured light systems is sub-striperesolution. The stripe code T, which usually has granular nature due toquantization can be approximated by a continuous smooth surface, takinginto account the fidelity map. We used a separable cubic spline, whichwas fitted to the image T with weighting inverse proportional to pixelfidelity. As the result, a smooth stripe id image T with sub-striperesolution was obtained.

Let us denote by Bv and Bu the orthonormal spline bases corresponding tothe rows and the columns of T, respectively. Decomposition of the T inthe two-dimensional separable basis obtained as the tensor product of Byand Bu can be expressed as

C=B _(u) ^(T) TB _(c),  (37)

or, alternatively, as

c=B ^(T) t,  (38)

where t is the column-stack representation of T, c is a vector of splinecoefficients and B=B_(c)∅B_(u) is the Kronecker product of the row andthe column bases. Weighted spline fitting constitutes to finding suchspline coefficients c that minimize

$\begin{matrix}{{\sum\limits_{k}{\frac{1}{f_{k}}\left( {\left( {Bc} \right)_{k} - t_{k}} \right)^{2}}},} & (39)\end{matrix}$

where f_(k) denotes the fidelity of the pixel tk. We also add acontrollable penalty on irregularity of the smoothed image {tilde over(t)}=Bc. In matrix notation, the weighted spline fitting problem reads

c=argmin{∥WBc−Wt∥ ₂ ² +λ∥DBc∥ ₂ ²},  (40)

where

$W = {{diag}\left\{ \frac{1}{f_{k}} \right\}}$

is the weighting matrix, D is the matrix defining the irregularitypenalty and λ is the smoothness parameter, controlling the tradeoffbetween smoothness of {tilde over (t)} and faith to the original data.

The analytic solution for eq. (40) is

c=[B ^(T) B+λ(DB)^(T)(DB)]^(†) t,  (41)

where A^(†)=(A^(T)D)⁻¹ A denotes the Moore-Penrose pseudoinverse.

Since different amount of smoothing should be usually applied in thehorizontal and the vertical directions of T, it is reasonable to use twopenalty factors with two separate smoothness parameters, which controlthe smoothness in each direction. We used the L₂ norm of a finitedifference operator as the penalty factor, yielding

c=[B ^(T) B+λ _(x)(D _(x) B)^(T)(D _(x) B)+λ_(y)(D _(y) B)^(T)(D _(y)B)]^(†) t,  (42)

where D_(x) and D_(y) are the row-wise and the column-wise discretederivative operators and λ_(x), and λ_(y) are smoothness parameterscontrolling the smoothness of {tilde over (T)} in the x- andy-direction, respectively. The resulting smooth stripe id image {tildeover (T)} is given (in column stack representation) by {tilde over(t)}=Bc.

FIG. 9A-B presents the T image of a human face obtained by directlydecoding the stripe ids (FIG. 9A) and by using fidelity-weighted smoothcubic spline fitting (FIG. 9B). Note that quantization noise is lesssignificant in the latter case.

FIG. 10A-C depicts the reconstructed surface with (FIG. 10B) and without(FIG. 10A) using sub-stripe resolution. FIG. 10C shows a verticalprofile of the Z-map without using sub-stripe resolution (solid) andusing sub-stripe resolution (dashed).

Calibration

In an embodiment, to find the camera and the projector matrices, theautomatic calibration procedure described hereinabove was used. As thecalibration object, a precisely built 15×15×15 cm wooden pyramidattached with 3 mutually perpendicular planes attached to a “background”plane, was used. Object surfaces were made nearly Lambertian using whitemate finishing.

228 circular fiducial points were marked on the calibration objectsurfaces. The points were grouped into sets of collinear equally-spacedmarks, 3 sets of 33 points each on the background plane, and two sets of22 points each on each of the pyramid sides (FIG. 11 ). WCS was definedas the local coordinate system of the calibration object.

At the first stage, the calibration object was scanned and a sub-striperesolution stripe id image {tilde over (T)} was calculated from the rawdata. At the second stage, the full-illumination image IH was binarized,the fiducial points were automatically located and their centroidcoordinates were calculated. A set of 228 stripe ids at locationscorresponding to the fiducial point centroids, together with thecentroid locations themselves was used as the input for the calibrationalgorithm. For numerical stability, WCS coordinates of the calibrationobjects, the image plane coordinates of the camera and the projectorstripe ids were normalized to [−1, 1].

First, camera and projector PPMs that minimize the forward projectionerror were found by solving eq. (31). These matrices were used as theinitial point of the Newton algorithm, which converged to the solutionof eq. (33), yielding camera and projector matrices, which minimize thebackprojection error. FIG. 12A-B depicts the forward projection of thetheoretically calculated calibration object fiducial points, onto thecamera and the projector coordinate system. The measured fiducial pointscentroids and projector stripe ids are shown as a reference. FIG. 12Ashows the object points reprojected to the camera image plane (circles)and the measured fiducial point centroids (crosses). FIG. 12B shows theobject points reprojected to the projector coordinate system (dashed)and the measured stripe ids (black). In both FIG. 12A and FIG. 12B, thecoordinates are normalized.

Table 1 shows the RMS and the maximum reconstruction errors of thecalibration object in five tests with random camera and projectorlocations. Two cases are studied: 1) when the camera and the projectormatrices are obtained by minimizing the forward projection error and 2)when the camera and the projector matrices are obtained by minimizingthe backward projection error. The errors were calculated on a set of100,000 points, with the analytical plane equations of the calibrationobject serving as a reference. Table 2 shows the improvement of the RMSand the maximum reconstruction error when using the optimalbackprojection instead of the optimal forward projection. RMS wasimproved in all tests (improvement ranging from 1.44% to 45.66%, 12% onaverage). Maximum error was improved in all tests except Test 2, wherethe maximum error obtained using the optimal backprojection worsened by6.58% (the fact that improvement in the RMS error was observed in Test 2suggests that the degradation in the maximum error might have beencaused by a spurious pixel). The maximum improvement, about 50%, wasobtained in Test 5.

TABLE 1 RMS and maximum errors for reconstruction of the calibrationobject using the optimal forward projection and the optimal backwardprojection. Optimal projection Optimal backprojection Test RMS (mm) Max.(mm) RMS (mm) Max. (mm) 1 0.0624 0.2790 0.0577 0.2295 2 0.0626 0.28810.0616 0.3084 3 0.0423 0.2054 0.0417 0.1723 4 0.1599 0.6823 0.15560.6337 5 0.1579 0.6170 0.1084 0.4122

TABLE 2 Improvement in the RMS and the maximum error in reconstructionof the calibration object when using the optimal backprojection insteadof the optimal forward projection. Test RMS Max. 1 8.15% 21.57% 2 1.62%−6.58% 3 1.44% 19.21% 4 2.76%  7.67% 5 45.66%  49.68%

The RMS error was about 0.5 mm in Tests 1-3 and about 1.6 mm in Tests4-5. The latter degradation of the reconstruction quality was due to thehighly oblique position of the calibration object with respect to thecamera, which resulted in lower SNR, since less light was reflected fromthe object planes.

FIG. 13A-D shows two profiles of the reconstructed calibration object.Analytical object profiles are shown as a reference. FIG. 13A shows thereconstructed profile along the solid line in FIG. 13B, while FIG. 13Cshows the reconstructed profile along the solid line in FIG. 13D. InFIGS. 13A and 13C, the solid line shows the reconstruction using theoptimal forward projection and the dash-dotted line shows thereconstruction using the optimal backprojection. The dashed line showsthe real profile. All units are in cm.

FIG. 14 illustrates an embodiment of the system (1100). In thisembodiment, the system comprises a PC (1140) controlling a camera unit(1115) and modulation unit (1135) for spatial and temporal modulation ofthe light source, a light source (1130), a camera (1110) and anendoscope (1110).

The camera unit (1115) controls the imaging system, including, wherenecessary, focusing of the camera (1110) and, where necessary,manipulation of the system optics so that the desired portion of thescene is at the center of the field of view. Images are transmitted fromthe camera (1110) to the PC (1140) via the camera unit (1115).

The modulation unit (1135) controls positioning of the light (1132),both spatially and temporally, in order to provide a set of structuredlight rays as described hereinabove. The modulation unit (1135) can be“upstream” of the light source (1130), as shown, or “downstream” of thelight source (1130).

In embodiments where the modulation unit (1135) is “upstream” of thelight source (1130), the modulation unit (1135) controls the lightsource (1130) directly so that light (1132) is emitted from differentpositions in the source (1130) at different times, in order to providethe desired set of structured light rays.

In preferred embodiments, where the modulation unit is “downstream” ofthe light source (1130), the modulation unit (1135) controls positioningof the light beam (1132); light from the light source (1130) ismaneuvered by the modulation unit (1135) to provide the desired set ofstructured light rays.

Software to carry out the functions of the modulation unit (1135) andthe camera unit (1115) can be local, within the unit, or central, withinthe PC (1140).

The structured light rays are then transmitted through the endoscope(1120) so as to illuminate the region of interest.

Reflected radiation from the objects in the region of interest istransmitted back through the endoscope (1120) so as to be received bythe camera (1110), and the camera image is transmitted via the cameraunit (1115) to the PC, where the 3D image of the region of interest isreconstructed, as described hereinabove. Manipulation of the cameraimage, such as elimination of background light or binarizing, asdescribed hereinabove, can be carried out in the camera unit (1115), inthe PC (1140), or any combination thereof.

FIG. 15 shows an embodiment of a distal end (1000) of the endoscope.Modulated light (1132), after transmission through the endoscope, isemitted by the endoscope's optical transmission channels. Afterreflection from an object (not shown) it is received by the distal endof the camera optics (1020). The distal end of the camera opticscomprises an optical element (1025), typically a lens or prism. Thisoptical element (1025) is at an angle to the laparoscope other than 90degrees, since, as described hereinabove, the optic axis of the cameracan not be parallel to the plane of the structured light. In preferredembodiments, the angle between the plane of the structured light raysand the normal to the face of the distal optical element is betweenabout 30 degrees and about 60 degrees.

Example 2

In some embodiments, the physical setup is similar to that of theembodiment of Example 1 (see FIG. 1 ) and, as in the embodiment ofExample 1, a calibration object of known shape and size (the worldobject) is illuminated by a set of rays of light (the projector, in theprojector coordinate system) emitted from known positions, in knowndirections and at known times. These rays are reflected from thecalibration object (the object, in the world coordinate system) andgenerate at least one 2D camera image (the camera image, in the cameracoordinate system).

However, instead of the operators of Example 1, polynomial fits are madeto the known spot positions on the camera images, given the known raypositions, the known spot positions on the camera images and the knownshape and size of the calibration object. From the polynomial fits, thelocations of positions on the surface of an unknown object can be found.

In the embodiment of Example 2, the projector (light source) projects acoded stripe pattern on the object or objects to be viewed and thecamera captures an image of the reflected light. Hence, for each visiblepoint in the scene there is a corresponding stripe number and imagelocation (in pixel coordinates). Given the parameters of the system,each strip value defines a plane in the world (the space being viewed)and each pixel defines a ray in the world. The intersection of the planeand the ray defines a unique 3D location in the world. The parameters ofthe system are obtained during calibration. As in Example 1, acalibration object is placed at a known position in the field of view.The calibration object comprises a set of fiducial marks whose spatiallocation is known to high accuracy. The system is operated normally andthe pixel coordinates are found for each of the fiducial marks. Thesetriples of fiducial mark, pixel and stripe coordinates are used toestimate the unknown parameters of a mathematical model of the system.

In practice, the projector projects a sequence of 9 banded lightpatterns onto the scene. The first pattern is full illumination. Theremaining patterns provide temporal encoding, with each of the 8patterns providing one bit in an 8 bit binary grey code for the stripevalue. In general, for a b bit code, the k^(th) bit (where k=0, 1, . . ., b−1) of the code for the n_(th) value (n=0, 1, . . . , 2^(b-1)) isgiven by

$\left\lbrack {\frac{1}{2}\left( {\frac{n}{2^{k}} + 1} \right)} \right\rbrack$

mod 2. An important property of the code is that no two edges (levelchanges) are coincident throughout the sequence. As the most likelyplace to erroneously read a bit is at a level change, the code helpsreduce the possibility of detecting more than one bit of the codeerroneously. In addition, detecting a bit incorrectly at a level changeonly alters the stripe value by one. It should be noted that this stripeencoding completely eliminates the need for resolving any correspondencebetween points in the image and stripes.

Projector lens distortion results in each stripe forming a slightlycurved surface instead of a plane, and camera lens distortion results inthe image point being slightly displaced from its nominal position.Furthermore, the discrete nature of pixel-based cameras and projectorsgives rise to uncertainty in the true pixel coordinates and stripevalue.

In preferred variants of the present embodiment, correcting for theeffects of projector lens distortion is included in the model of theprojector.

In preferred variants of the present embodiment, correcting for theeffects of camera lens distortion is via subpixel and substripeoperators.

In the present embodiment, the uncorrected transformation of a point Pfrom the world coordinate system C_(w) to the camera coordinate systemC_(c) is

$\begin{matrix}{\begin{pmatrix}x_{c} \\y_{c} \\z_{c}\end{pmatrix} = {{R_{c}\begin{pmatrix}x_{w} \\y_{w} \\z_{w}\end{pmatrix}} + T_{c}}} & (43)\end{matrix}$

where X_(c)=(x_(c), y_(c), z_(c)) is the location of the point in cameracoordinates, X_(w)=(x_(w), y_(w), z_(w)) is the location of the point incamera coordinates, R_(c) is a 3×3 rotation matrix and T_(c) is a 3×1translation vector.

The “principal point” is the intersection of the imaging plane with theoptical axis. Without loss of generality, a 2D image coordinate systemcan be defined as being in the image plane with its origin located atthe principal point. Let p be the projection of the point P onto theimage plane for a non-distorting projector and let the coordinates of pbe ({tilde over (X)}_(i), {tilde over (y)}_(i), 0). Then

$\begin{matrix}{\begin{pmatrix}{\overset{\sim}{x}}_{i} \\{\overset{\sim}{y}}_{i}\end{pmatrix} = {\frac{f_{c}}{z_{c}}\begin{pmatrix}x_{w} \\y_{c}\end{pmatrix}}} & (44)\end{matrix}$

where f_(c) is the “principal distance”.

As described above, the projective lenses slightly distort the plane ofthe projected light into a curved surface. To correct for this, let(x_(i), y_(i), 0) be the observed location in the image plane of thepoint p, after projection by a real, distorting projection system. Then,

$\begin{matrix}{\begin{pmatrix}x_{i} \\y_{i}\end{pmatrix} = {\begin{pmatrix}{\overset{\sim}{x}}_{i} \\{\overset{\sim}{y}}_{i}\end{pmatrix}\left\lbrack {1 + {K_{c}\left( {{\overset{\sim}{x}}_{i}^{2} + {\overset{\sim}{y}}_{i}^{2}} \right)}} \right\rbrack}} & (45)\end{matrix}$

where K_(c) is a coefficient determined by the amount of radialdistortion.

The effects of pixels size and imager speed can also be factored in. LetC_(pixel) be a pixel coordinate system associated with the digitalimage, where the location of a point is given by (x,y). Then the pixelcoordinates are related to the image coordinates by

$\begin{matrix}{\begin{pmatrix}x \\y\end{pmatrix} = {{\begin{pmatrix}s_{c}^{x} & k_{c} \\0 & s_{c}^{y}\end{pmatrix}\begin{pmatrix}x_{i} \\y_{i}\end{pmatrix}} + \begin{pmatrix}c_{c}^{x} \\c_{c}^{y}\end{pmatrix}}} & (46)\end{matrix}$

where S_(c) ^(x) and S_(c) ^(y) are scale factors in pixels/mm, C_(c)^(x) and C_(c) ^(y) are the pixel coordinates of the principal point andk_(c) is a shear coefficient in pixel/mm.

The model for the projector is similar to that for the camera, as theprojector can be regarded as a camera with a 1D image, acting inreverse.

However, the projector model differs from the camera model in that thestripe coordinate system C_(s) is 1D, unlike the 2D camera coordinatesystem C_(p), leading to the equation:

x _(s) =s _(p) ^(x) x _(i) +c _(p) ^(x)  (47)

Where s_(p) ^(x) is a scale factor and c_(p) ^(x) is the pixelcoordinate of the principal point of the projection system.

Physically, each stripe value x_(x) gives rise to a unique line on theprojector given by {(x_(i),y_(i)):x_(i)=(x_(s)−c_(p) ^(x))/s_(p) ^(x),y_(i)∈

}. The line is projected and distorted by the lens, giving rise to aslightly curved surface in the world.

Altogether, the following set of equations describes thetransformations:

$\begin{matrix}{\begin{pmatrix}x_{c} \\y_{c} \\z_{c}\end{pmatrix} = {{R_{c}\begin{pmatrix}x_{w} \\y_{w} \\z_{w}\end{pmatrix}} + T_{c}}} & (43)\end{matrix}$ $\begin{matrix}{\begin{pmatrix}{\overset{\sim}{x}}_{i} \\{\overset{\sim}{y}}_{i}\end{pmatrix} = {\frac{f_{c}}{z_{c}}\begin{pmatrix}x_{w} \\y_{c}\end{pmatrix}}} & (44)\end{matrix}$ $\begin{matrix}{\begin{pmatrix}x_{i} \\y_{i}\end{pmatrix} = {\begin{pmatrix}{\overset{\sim}{x}}_{i} \\{\overset{\sim}{y}}_{i}\end{pmatrix}\left\lbrack {1 + {K_{c}\left( {{\overset{\sim}{x}}_{i}^{2} + {\overset{\sim}{y}}_{i}^{2}} \right)}} \right\rbrack}} & (45)\end{matrix}$ $\begin{matrix}{x_{i} = {{s_{p}^{x}x_{i}} + c_{p}^{x}}} & (47)\end{matrix}$

The above models describe the geometric behavior of the camera andprojector and are based on a physical understanding of these devices.K_(c)=0 and k_(c)=0 describe a perfect camera. Similarly, K_(p)=0describes a perfect projector.

It is clear that there is a redundant parameter within the set of cameraparameters {s_(c) ^(x), s_(c) ^(y), f_(c)}. This can be dealt with byarbitrarily fixing one of them. For the calibrations describedhereinbelow, f_(c) is set to 1.

Lens distortion has been incorporated as a function of the transformfrom ({tilde over (x)}_(i), {tilde over (y)}_(i), 0) to (x_(i), y_(i),0). This is because the distortion is a 2D phenomenon whereas only onecoordinate of the position of a point on the projector emitter isavailable. However, the ranges of both formulations are almostequivalent for typical system configurations. Therefore, thismodification makes little difference to the solution.

The camera and projector models can be summarized as follows. Let

Θ_(c)=(s _(c) ^(x) s _(c) ^(y) c _(c) ^(x) c _(c) ^(y) k _(c) K_(c)ω_(c) ¹ω_(c) ²ω_(c) ³ T _(c) ¹ T _(c) ² T _(c) ³)^(t)  (48)

Θ_(p)=(s _(p) ^(x) c _(p) ^(x) K _(p)ω_(p) ¹ω_(p) ²ω_(p) ³ T _(p) ¹ T_(p) ² T _(p) ³)^(t)  (49)

where w_(c) ^(k), k=1 . . . 3 and ω_(n) ^(k), k=1 . . . 3 parameterizeR_(c) and R_(p) respectively (e.g. Euler angles). The first sixparameters of the camera model and the first three parameters of theprojector model are referred to as “intrinsic parameters” because theyare independent of the world coordinate frame.

Using the above, the transformation from world coordinates p_(w) topixel coordinates p_(p) can be written as

p _(p) =F _(c)(p _(w);Θ_(c)).  (50)

Similarly, the transformation from world coordinates to the stripe valuep_(x) can be written as

p _(s) =F _(p)(p _(w);Θ_(p)).  (51)

Both the calibration and spatial intersection procedures require thepixel coordinates and the stripe value of points of interest in theworld (field of view). Obtaining good estimates for the value of thesequantities is important in order to obtain accurate estimates of theparameters of the system during calibration and accurate estimates of 3Dpoint locations during spatial intersection.

Subpixel Estimation

During calibration, the subpixel locations of the centroids of thefiducial marks on the calibration reference are required. The grayscalecentroid operator can be used to estimate the centroids of the fiducialmarks in the image. Such centroid-based techniques have the followingadvantages:

-   -   They are robust to noise, since they are based on integration        rather than the differentiation of many edge detectors.    -   They are robust to small amounts of aliasing. Theoretically,        they are exact if the sampling frequency is greater than the        Nyquist frequency.    -   They are computationally inexpensive.

However, the centroid of the image of a fiducial mark does not ingeneral coincide with the image of the centroid, due to perspectivedistortion. This introduces a systematic error in the centroid estimateand is a fundamental limitation of centroiding techniques.

Ignoring lens distortion, an estimate of this error can be obtained asfollows. Consider a flat circular region Δ of radius r. let n be thenormal to Δ and q be the location of the centroid of Δ, both in cameracoordinates. If c denotes the image of the centroid and ĉ denotes thecentroid of the image of Δ then

$\begin{matrix}{c = {\frac{f_{c}}{t_{3}}\begin{pmatrix}t_{1} \\t_{2}\end{pmatrix}}} & (52)\end{matrix}$ $\begin{matrix}{\hat{c} = {\frac{f_{c}}{t_{3}^{2} + {r^{2}\left( {n_{3}^{2} - 1} \right)}}{\begin{pmatrix}{{t_{1}t_{3}} + {r^{2}n_{1}n_{3}}} \\{{t_{2}t_{3}} + {r^{2}n_{2}n_{3}}}\end{pmatrix}.}}} & (53)\end{matrix}$

As required, ĉ reduces to c when n=e₃ (Δ parallel to the image plane.The error in the image plane is given by e_(p)=c−ĉ. This error can betransformed into pixel coordinates using the scale factors and shearcoefficient defined hereinabove (eq. 46).

FIG. 16 shows a plot of these errors in pixel coordinates for a typicalcalibration configuration. Each error is represented as a vector withits base at the associated fiducial mark. The vector magnitudes havebeen scaled by a factor of 1000 for clarity. In the current system,these errors are considered acceptable.

During special intersection, two situations can arise, one where theimage location is freely chosen and the other where it is dictated by afeature in the world. The need to use subpixel estimation does not arisein the first situation, e.g., during generation of a dense range mapwith image locations at the pixel lattice points. The type of subpixeloperator used during the second situation depends on the nature offeatures (such as, but not limited to edges, ridges, blobs, voids, ortransparent or translucent regions).

Substripe Estimation

During calibration and special intersection, the substripe values ofspecified world points are required.

Substripe estimation is more difficult than subpixel estimation. Onereason for this is that the stripe information is not directly availablefrom the projector in the sense that the pixel location information isavailable from the camera. It is the images of the stripes cast on thescene which are used to recover the stripe values. In this way, thestripes are effectively samples twice, once by the predeterminedlocations of the stripes emitted by the projector, and once from theimage in the camera. Another difficulty, specific to temporally encodedsystems, is that the stripe values are encoded in a sequence of imagesrather than available in a single image as for pixel locations.

It is assumed that the 8 bit stripe value is available for each pixel inthe image. The image formed from these values is referred to as a“stripe image”. The discrete nature and coaresse quantization of thestripe image gives rise to poor accuracy if used directly. The algorithmadopted for substripe estimation is simple. A region of interest, Ω, isestablished with its center at the subpixel location, (x_(p),y_(p)),where the substripe estimate, x_(x), is to be determined. Aleast-squares polynomial facet is fitted to the stripe image in Ω andthe substripe estimate made by interpolating this facet.

If the underlying surface is flat, a planar facet is fitted to thestripe image Ω. Assuming no lens distortion in the camera and theprojector, the stripe values and the pixel coordinates for a planarpatch in the world will be related by a rational function because thecomposition of two projective maps is a projective map. Hence a planarfacet does not model perspective effects precisely. The errorsintroduced by using a planar facet can be estimated as follows.

Let C_(c) and C_(p) denote the 3×4 and 2×4 Perspective TransformationMatrices (PTMs) for the camera and projector, respectively. Both thecamera and the projector are assumed to be distortionless. Let T be a4×3 homogeneous transformation matrix describing the coordinatetransformation between a 2D coordinate system associated with the planarpatch and the world coordinate system. It follows that

$\begin{matrix}{{\alpha\begin{pmatrix}x_{s} \\1\end{pmatrix}} = {C_{p}{T\left( {C_{c}T} \right)}^{- 1}\begin{pmatrix}x_{p} \\y_{p} \\1\end{pmatrix}}} & (54)\end{matrix}$

This shows that the stripe values and pixel coordinates are related byan equation of the form

$\begin{matrix}{x_{x} = \frac{{n_{1}x_{p}} + {n_{2}y_{p}} + n_{3}}{{d_{1}x_{p}} + {d_{2}y_{p}} + d_{3}}} & (55)\end{matrix}$

Let x_(x) be the true substripe value at subpixel location (x_(p),y_(p))calculated using eq. (54). Let (X_(p),Y_(p)) be the n×2 matrix of pixelcoordinates in Ω and X_(x) be the n vector of substripe values atcoordinates (X_(p),Y_(p)). The 3×1 vector of coefficients of the leastsquares plane fitted to Xx on Ω is given by

c=(A′A)⁻¹ A′X _(s)  (56)

where A=[X_(p) Y_(p) 1_(n)] is the n×3 design matrix associated with theplane. The estimate of the x_(x) is given by

{circumflex over (x)} _(s) =c ₁ x _(p) +c ₂ y _(p) +c ₃  (57)

and the error is given by

FIG. 17 shows a stem plot of these errors for a typical calibrationconfiguration. The horizontal axis gives the stripe value calculated atthe centroids of the fiducial marks and the vertical lines give theerror in stripes at these locations. The errors are insignificantcompared with the effects which result from the 8 bit quantization ofthe stripe value and therefore can typically be ignored.

In the more general case of a curved surface, higher order 2D polynomialfacets are fitted to the stripe image. The order of the facet modeldepends on the size of the region of interest Ω, which must cover asufficient number of stripes. Typically, a region of interest of size17×17 pixels and 2D third-order polynomials are used. Note thatpolynomial fitting does not introduce blurring in the sense of low-passfiltering. For example a polynomial of degree n (which can haveconsiderable oscillation) is invariant to a polynomial filter of order n(i.e. the filter which is defined by fitting and interpolating apolynomial of degree n).

Planar facets are only used when the surface under consideration isknown to be flat, such as the faces of the calibration reference. In allother cases. a polynomial facet is used. The planar facet is embodiedbecause it provides superior estimation when the surface has very lowcurvature.

Calibration

Calibration involves estimating the unknown parameters relating theworld points, the projector points and the image points from a number ofknown world points and their corresponding pixel coordinates and stripevalues.

For a calibration object with n fiducial marks, let p_(w)(j), p_(p)(j)and p_(s)(j) be the world coordinates, the pixel coordinates and thestripe coordinates, respectively, of the jth fiducial mark. In addition,let P_(w), P_(p) and P_(s) be vectors of sizes 3n, 2n and n,respectively, formed by stacking the coordinate vectors for the worldpoints, the pixel points and the stripe points, respectively. Thevectors P_(p) and P_(s) are measured from the image sequence asdescribed hereinabove and will consequently suffer from measurementnoise.

If μ_(P) _(p) and μ_(P) _(s) are the true values of P_(p) and P_(s),respectively, then the observations are given by

P _(p)=μ_(P) _(p) +ϵ_(p)  (58)

P _(s)=μ_(P) _(s) +ϵ_(s)  (59)

and the model can be written as

μ_(P) _(p) −F _(c)(P _(w);Θ_(c))=0  (60)

μ_(P) _(s) −F _(p)(P _(w);Θ_(p))=0  (61)

In this derivation, the assumptions are:

-   -   P_(w) is measured without error.    -   P_(p)˜N(μ_(P) _(p) , σ_(p) ²I_(2u))    -   P_(s)˜N(μ_(P), σ_(s) ²I_(n))

Maximum Likelihood Estimation of {Θ_(c), Θ_(p)} leads to the non-linearleast squares (NLLS) problem

$\begin{matrix}{\min\limits_{\Theta_{c}}{{P_{p} - {F_{c}\left( {P_{w};\Theta_{c}} \right.}}}^{2}} & (62)\end{matrix}$ $\begin{matrix}{\min\limits_{\Theta_{p}}{{P_{s} - {F_{p}\left( {P_{w};\Theta_{p}} \right.}}}^{2}} & (63)\end{matrix}$

which can be solved using any general NLLS solving algorithm known inthe art. Non-limiting examples of such algorithms are Gauss-Newton orLevenberg-Marquardt.

Obtaining good initial estimates for the parameters is desirable as (1)it helps ensure that the global minimum is obtained and (2) reducescomputation by reducing the number of iterations required. Initialestimates can be obtained from a variety of sources, depending on thenature of the problem. A typical approach is to estimate and decomposethe PTMs for the camera and the projector.

Spatial Intersection

Special intersection is used to find an estimate of the location of apoint in the world coordinate system given its pixel coordinates andstripe value. The lens distortion in the projector results in slightlycurved surfaces being projected onto the scene rather than planes.Consequently, the special intersection procedure involves solving ahomogeneous non-linear system in three variables. However, linearmethods can be used to calculate a good initial approximation andphysical considerations preclude the presence of any other solutions ina large neighborhood of the true solution.

Let p_(p)=(x_(p),y_(p))^(t) and p_(x)=(x_(s)) be the pixel coordinatesand stripe value of a point P. Then the world coordinates of P, denotedp_(w), are given by the solution of the non-linear system

$\begin{matrix}{{\begin{pmatrix}p_{p} \\p_{s}\end{pmatrix} - \begin{pmatrix}{F_{c}\left( {p_{w};\Theta_{c}} \right.} \\{F_{p}\left( {p_{w};\Theta_{p}} \right.}\end{pmatrix}} = 0} & (64)\end{matrix}$

This problem can be solved using any general technique known in the art,for non-limiting example, a quasi-Newton strategy.

The initial linear estimate of p_(w) is found as follows. Let C_(c) ^(k)be the kth row of the 3×4 camera PTM and C_(p) ^(k) the kth row of the2×4 projector PTM. Then the PTM equations relating p_(w), p_(p) andp_(x), i.e.,

$\begin{matrix}{{{\beta_{p}\begin{pmatrix}p_{p} \\1\end{pmatrix}} = {C_{c}\begin{pmatrix}p_{w} \\1\end{pmatrix}}}{{\beta_{s}\begin{pmatrix}p_{s} \\1\end{pmatrix}} = {C_{p}\begin{pmatrix}p_{w} \\1\end{pmatrix}}}} & (65)\end{matrix}$

can be rearranged into the 3×4 linear homogeneous system

$\begin{matrix}{{\begin{pmatrix}{C_{c}^{1} - {x_{p}C_{c}^{3}}} \\{C_{c}^{2} - {y_{p}C_{c}^{3}}} \\{C_{p}^{1} - {x_{s}C_{p}^{2}}}\end{pmatrix}\begin{pmatrix}{\alpha p_{w}} \\\alpha\end{pmatrix}} = 0} & (66)\end{matrix}$

A solution to eq. (66) can be obtained (for non-limiting example) usingSVD and the estimate of pw obtained from the homogeneous solution to eq.(66).

When there are a large number of points to process, computationalsavings can be made. A generalization of the vector cross-produce forn−1 vectors in an n-dimensional linear space can be defined implicitlyby

w, v₁× . . . ×v_(n-1)

=det(v₁, . . . v_(n-1), w). v₁× . . . ×v_(n-1) can be written asΣ_(k)det(v₁, . . . v_(n-1), e_(k))e_(k) and is orthogonal to v_(k) where{ek}_(k=1) ^(n) are the standard basis vectors of

^(a). Therefore, a solution to eq. (66) is given by the (generalized)cross product of the rows of the 3×4 matrix

$\begin{matrix}{\begin{pmatrix}{\alpha p_{w}} \\\alpha\end{pmatrix} = {\sum\limits_{k = 1}^{4}{\det\left( {{C_{r}^{1} - {x_{p}C_{r}^{3}}},{C_{r}^{2} - {y_{p}C_{r}^{3}}},{C_{p}^{1} - {x_{s}C_{p}^{2}}},e_{k}} \right)e_{k}}}} & (67)\end{matrix}$

Dividing by α and using the linearity and antisymmetry of thedeterminant tensor, the kth component of p_(w) is given by

$\begin{matrix}{p_{w}^{k} = \frac{\begin{matrix}{C_{1,2,1}^{k} - {x_{p}C_{3,2,1}^{k}} - {y_{p}C_{1,3,1}^{k}} -} \\{{x_{s}C_{1,2,2}^{k}} + {x_{s}x_{p}C_{3,2,2}^{k}} + {x_{s}y_{p}C_{1,3,2}^{k}}}\end{matrix}}{\begin{matrix}{C_{1,2,1}^{4} - {x_{p}C_{3,2,1}^{4}} - {y_{p}C_{1,3,1}^{4}} -} \\{{x_{s}C_{1,2,2}^{4}} + {x_{s}x_{p}C_{3,2,2}^{k}} + {x_{s}y_{p}C_{1,3,2}^{4}}}\end{matrix}}} & (68)\end{matrix}$

where C_(i,j,l) ^(k)=det(C_(c) ^(i), C_(c) ^(j), C_(p) ^(l), e_(k)) areconstants which depend only on the camera and projector PTMs and can beprecomputed.

In these experiments, the system comprised a K2T LCS and controllerhoused in a Kodak projector. TM-6CN Pulnix camera and S2200 Data Cellframegrabber. The LCS shutter has 256 stripes and 512×512 images werecaptured. The calibration reference is a 150 mm cube with 72 fiducialmarks of 5 mm radius arranged over 3 faces, as shown in FIG. 18 . The 3Dlocation of each fiducial mark centroid has been measured to an accuracyof 0.1 mm in a coordinate system attached to the cube. Duringcalibration, this coordinate system is taken to be the world coordinatessystem. Calibration thus fixes the world coordinate system and allsubsequent measurements are expressed in the world coordinates system.The experimental setup has the projector above the camera with the anglebetween their optic axes being approximately 12° to give a workingvolume of 250 mm diameter at 1600 mm in front of the camera.

To evaluate the performance of the system, seven trials were used withthe calibration reference in a different position in each. For eachtrial, the observed data consists of the pixel coordinates and stripevalues of the fiducial marks. The calibration parameters for each trialwere extracted from the observed data using the procedure describedhereinabove.

The estimated pixel coordinates and stripe values are obtained byprojecting the world reference coordinates using the calibrated systemmodel. The pixel residuals (FIG. 19 ) and stripe residuals (FIG. 20 )are the difference between the measured and estimated values. Thespatial intersection errors (FIG. 21 ) are the difference between thereference coordinates of the fiducial marks and their coordinatesestimated by special intersection. FIGS. 19, 20 and 21 show the pixelresiduals, stripe residuals and spatial intersection errors for trial 4.The fiducial mark numbers in FIG. 21 were obtained from an arbitraryordering of the fiducial marks on the cube.

Table 3 shows the average magnitude of the special intersection errorsfor each of the calibration trials. Each average was obtained using all72 fiducial marks. For comparison, Table 4 shows the average errors whenradial distortion is neglected for the camera and projector models.Table 5 shows the average errors when no substripe estimation is used.Excluding either radial distortion or substripe estimation increases theerror in all trials. On average, over all trials, adding substripeestimation improves the performance by 82% while adding lens distortionimproves the performance by only 13%, showing that substripe estimationhas the more significant effect.

TABLE 3 Average magnitude of the spatial intersection error forcalibration trials. Trial 1 2 3 4 5 6 7 average error (mm) 0.154 0.2450.296 0.187 0.280 0.246 0.233

TABLE 4 Average magnitude of the spatial intersection error forcalibration trials if radial distortion is neglected. Trial 1 2 3 4 5 67 average error (mm) 0.226 0.283 0.311 0.223 0.292 0.261 0.264

TABLE 5 Average magnitude of the spatial intersection error forcalibration trials with no substripe estimation. Trial 1 2 3 4 5 6 7average error (mm) 1.348 1.312 1.371 1.521 1.233 1.170 1.444

Given the observed data for a trial and the calibration parameters forthat trial (or any other), special intersection can be used to estimatethe coordinates of the fiducial marks in the world coordinate systemestablished by the calibration. These can be compared with the referencecoordinates by estimating, using (for non-limiting example) a leastsquares error criterion, the 3D rigid body transformation needed totransform the reference coordinates to the estimated coordinates (modelfitting). The model fitting error is the difference between theestimated world coordinates and the transformed reference coordinates.Table 6 shows the average magnitude of the model fitting error for allof the calibration trials and all of the observed data sets. Thediagonal entries of Table 6 are very close to, and less than, the valuesin Table 3. This indicates that the spatial intersection procedure isaccurately recovering the coordinates in the world coordinate system.The model fitting errors are largest when the spatial intersection dataset is farthest from the calibration data set.

TABLE 6 Average magnitude of the model fitting error (in mm) for eachcalibration trial (rows) and each measured data set (columns). trial 1 23 4 5 6 7 1 0.154 0.405 0.271 0.207 0.322 0.285 0.298 2 0.389 0.2450.445 0.337 0.246 0.428 0.392 3 0.335 0.613 0.296 0.367 0.490 0.3070.437 4 0.205 0.379 0.312 0.187 0.298 0.297 0.248 5 0.408 0.301 0.4760.349 0.280 0.451 0.374 6 0.324 0.564 0.273 0.355 0.455 0.245 0.424 70.295 0.475 0.308 0.247 0.366 0.292 0.233

Table 7 shows the intrinsic parameters for the system obtained from eachtrial. The camera scale factors obtained during calibration agree withthe nominal value of 2200 pixels/mm. In addition, the camera principalpoint is consistent with the nominal value of (255,255) pixels. Theshear coefficient is small with respect to s_(c) ^(x) (0.14%),suggesting a very small contribution. The scale factor for the projectoris close to the nominal value of 120 stripes/mm.

TABLE 7 The 6 intrinsic parameters for the camera (upper) and the 3intrinsic parameters for the projector (lower) for each calibrationtrial. trial 1 2 3 4 5 6 7 s_(c) ^(x) 2411.062 2399.667 2425.2552418.480 2403.901 2443.054 2408.316 s_(c) ^(y) 2412.349 2402.9822425.084 2421.023 2407.875 2445.911 2410.448 s_(c) ^(x) 289.389 283.201301.069 274.053 271.295 314.980 272.129 c_(c) ^(y) 271.802 303.288258.478 275.559 309.616 261.080 259.293 k_(c) 4.211 2.316 4.308 3.2851.938 4.006 3.136 K_(c) 0.464 0.176 0.449 0.439 0.200 0.219 0.035 s_(p)^(x) 899.157 1020.565 1151.871 1015.006 1061.455 1162.614 1059.352 c_(p)^(x) 159.906 240.831 130.128 183.441 223.480 132.600 181.811 K_(p) 0.0850.123 0.317 0.116 0.138 0.384 0.409

However, there are significant variations in many of the parameters. Anexamination of the estimated dispersion matrix for the camera shows someparameters have high variances and there are high correlations betweenparameters. This can be partially explained by the experimentalconfiguration of the structured light system; with the distance betweenthe camera and the world reference much larger than the diameter of theworld reference a weak perspective camera model provides a reasonableexplanation of the observed data.

Consequently many parameters are correlated. In particular, has arelatively large variance and is highly correlated with s_(c) ^(x) ands_(c) ^(y).

The above remarks are also applicable to the projector parameters.However, the situation for eh projector is inherently worse than thatfor the camera as the projector has nine parameters (compared to 12 forthe camera) but only half as many data points are available. Asdescribed above, introducing K_(p) allows T_(p) ² to be estimated, but,as expected, it has a large variance.

Furthermore, when the distance between the projector and the worldreference is much larger than the diameter of the world reference, thestripe planes are nearly parallel in the working volume. Hence acomponent of the rotation is nearly unidentifiable. This gives rise tolarge variances in the rotation parameters.

It is important to note that the actual values of the camera andprojector parameters are incidental in this situation and it is thevalues of the measured 3D coordinates that are of interest. Therefore,including T_(p) ² and K_(p) in the projector model is useful because itimproves the accuracy of the spatial data, even though T_(p) ² is notparticularly accurate.

In preferred embodiments, optical fibers are used to transmit themodulated light through the endoscope to the region of interest, so thatthe effective area of the projector is minimized.

In preferred embodiments, calibration of the system is carried out whenthe system is produced. Preferably, no recalibration is needed. However,if recalibration is needed, it can be carried out on site, even in theoperating theater.

In preferred embodiments, a standard camera can be used, withpositioning of the camera field of view controlled by positioning theendoscope.

A further advantage of the system of the present invention is thatstandard systems can be used to transmit the modulated light to theregion of interest, for example, through the endoscope or laparoscope,or through a thoracostomy.

In preferred embodiments of the system, a standard endoscope is used. Inpreferred embodiments, the light source is a laser; the location fromwhich the light enters the working area can be altered so as to producethe stripe pattern needed for reconstruction disclosed above.

In some embodiments using the reconstruction method disclosed above, theaxis of the wide-angle lens is not parallel to the plane formed by thestripe pattern since as described above, reconstruction is not possiblewhen a ray between the camera focal point and the object point (parallelto the axis of the lens) is parallel to the plane originating at theprojector focal point and passing through the object point (the planeformed by the stripe pattern).

In some embodiments of the system, in addition to live-streaming theimages, the system can capture still images and store them in adatabase.

In some embodiments, the light source can be at least one spectral rangeselected from a group consisting of the visible, near infrared,infrared, or ultraviolet spectral regions.

Image transmission can be by using any of wireless communication,transmission of optical signals through fiber optic cables, ortransmission of electrical signals through wires or cables.

The light source can be a laser, Light Emitting Diodes (LED's) or anyother source known in the art. Preferably the illumination is insubstantially the same spectral range as that to which the camera issensitive.

In some embodiments, display of the 3D scene can be by any conventional3D display means such as, but not limited to, a 3D screen, 3D glasses,autostereoscopy, holography, a volumetric display, integral imaging,wiggle stereoscopy, an active shutter 3d system, a polarized 3d system,an interference filter system, a color anaglyph systems a Chromadepthsystem, an over/under format, and any combination thereof.

In preferred embodiments, the operating tools can be detected usingdepth cues.

In some embodiments, structures such as blood vessels, nerves, orcertain limbs can be detected by the system. In preferred variants ofsuch embodiments, the detected structures can be emphasized by any meansknown in the art such as, but not limited to, virtual reality overlays,color markings, pattern markings, outlines, arrows, and any combinationthereof.

In some embodiments, data from other modalities such as, but not limitedto, CT, MRI, ultrasound, PET and any combination thereof can bedisplayed to the surgeon. Preferably, the endoscope image and theimage(s) from the other modalit(ies) are matched to and aligned witheach other in 3D, so that the display includes all of the imageinformation.

In some embodiments, additional information such as recommendationsuseful in directing the progress of an operation, information useful inassisting an operator in making a decision, information useful for arecord of the operation and any combination thereof can be displayed tothe surgeon, either visually or audibly. This additional information canbe, but is not limited to, camera pose, body temperature, time, elapsedtime since start of operation, patient notes, notes made during theoperation, and other information accessible to the system and anycombination thereof.

Color Modulation

Color modulation can be used to distinguish different types of tissue,since different types of tissue show different patterns of scatteringand absorption at different wavelengths due to the different fractionsof e.g., lipids, deoxyhemoglobin (HHb), oxyhemoglobin (O₂Hb) and waterin the tissues. FIG. 22 shows absorption patterns for lipids,deoxyhemoglobin (HHb), oxyhemoglobin (O₂Hb) and water as a function ofwavelength for wavelengths from about 650 nm to about 1000 nm.

Color modulation can be used to identify hemodynamics and vascularreactivity, aiding in diagnosis of cancerous tissue and vasculardisease. For non-limiting example, in vascular disease, the restrictedblood flow to the tissue results in ischemia, hypoxia and tissuenecrosis, all identifiable by increased HHb and decreased O₂Hb.

An example of the difference between normal tissue and cancerous tissueis shown in FIG. 23 , in which, for 58 subjects, the absorption patternfor normal (2320) and cancerous (2310) breast tissue is compared. Theabsorption pattern between about 650 nm and 850 nm is primarily due toHHb and O₂Hb, whereas the absorption pattern between about 850 nm andabout 1000 nm is primarily due to H₂O and lipids.

FIG. 24 shows the difference between cancerous (dashed arrows) andnormal tissue (solid arrows) for 65 subjects. The curve for the normalsubjects is nearly flat, while the curve for the cancerous tissue showsa peak and trough characteristic of Hb, a lipid trough and water peak.

FIG. 25 shows the difference between cancerous (dashed arrows) andfibroadenomous tissue (dotted arrows) for 40 subjects. The peak andtrough characteristic of Hb, the lipid trough and the water peak aresignificantly larger for the cancerous than the fibroadenomous tissue.

Control of the Field of View

In preferred embodiments of the present system, the system comprises allthe mechanisms required to control fully the movement of both themovement of and the articulation of an articulated endoscope so that theposition of the endoscope and the position and angle of the tip of theendoscope are fully under control.

In preferred embodiments, movement of the endoscope and control of thearticulation of the endoscope are fully automatic; the surgeon indicatesa desired location and/or field of view and the endoscope movements andarticulation changes needed to achieve the desired location and/or fieldof view are carried out, while avoiding collisions with the tools and\orthe tissue, without need for further input from the operator.

Such control can be by means of a touchscreen, a joystick, voicecommands, eye movements, hand movements, or other control means underthe command of a surgeon.

Reference is now made to FIGS. 26 a and 26 b , which present in anon-limiting manner a system for maneuvering an endoscope (SFME) 130according to the present invention.

The system for maneuvering an endoscope 130 comprises at least onejoystick unit 152 and at least one communication means. At least one ofthe at least one communication means is adapted to connect at least onejoystick unit 152 to any standard maneuvering system 100 for maneuveringan endoscope 200. Different joystick units can control different aspectsof the endoscope's motion. For non-limiting example, a joystick unit tocontrol lateral movement of the endoscope, a joystick unit to controlzooming and a joystick unit to control articulation of the endoscopetip. Preferably, a single joystick controls all aspects of motion.

The communication means can comprise a wireless communication means, awired communication means and any combination thereof.

The connection between the joystick unit 152 and the maneuvering system100 (or other control system) enables control of the maneuvering system100 by the joystick unit 152, thereby enabling an operator to controlthe movement of the endoscope 200 by controlling the joystick unit 152.

Furthermore, the communication means is adapted to disconnect thejoystick unit 152 from the maneuvering system 100, thereby preventingthe endoscope 200 from being moved by the joystick unit 152.Disconnection of the joystick unit 152 prevents the situation ofunwanted movement of the endoscope 200 due to inadvertent joystick unit152 motion. Thus, in such a situation, movement of the joystick unit 152will not result in movement of the endoscope 120.

FIG. 26 b illustrates a closer view of the joystick unit 152. Uponpressing the joystick unit 152 in the direction of arrow 1521, theendoscope moves forward or backward. Upon pressing the joystick unit 152in the direction of arrow 1522, the endoscope moves left or right.

In preferred embodiments, movement of the endoscope is proportional tomovement of the joystick unit, unless the speed of the endoscope tip ina given direction would be above a predetermined value. In theseembodiments, movement of the endoscope at speeds greater than thepredetermined value is prevented. In these variants of the preferredembodiments, if a speed above the predetermined value is commanded, theendoscope will continue moving, but with a speed at or just below thepredetermined value. In some embodiments, if a speed above thepredetermined value is commanded, movement of the endoscope isprevented.

According to some embodiments of the present invention, the SFME 130 canbe wearable, either by a user or by an instrument. Reference is now madeto FIGS. 27 a-27 d which depict, in a non-limiting manner, a wearableoperator. FIGS. 27 a and 27 b depict the at least one joystick unit 152mounted in a operator 150, here a ring wearable by a user, while FIG. 27c depicts the at least one joystick unit 152 mounted on a operator 150attached to an exemplary surgical instrument and FIG. 27 d depicts aoperator 150 to be held in the hand. The operator 150 can be attached toany surgical instrument, can be attached to or mounted on the endoscope,or, as shown in FIG. 27 d , can be a free-standing unit which can siton, for example, the operating table, a side table or stand, or on ahospital bed.

Referring again to FIGS. 27 a-d , FIGS. 27 a and 27 c depict embodimentsof operator 150 with a single joystick unit 152, while FIGS. 27 b and 27d depict embodiments of the operator with two joystick units 152. Inpreferred embodiments of devices such as FIGS. 27 a and 27 c with asingle joystick unit, the joystick unit 152 controls maneuvering of theendoscope. In less-preferred embodiments, the joystick unit controlszoom of the endoscope. In embodiments of the operator 150 such as FIGS.27 b and 27 d with two joystick units, one joystick unit 152 controlsmaneuvering of the endoscope, while the other joystick unit 152 controlszoom of the endoscope.

Referring again to FIGS. 27 a-27 b , according to some embodiments, thewearable operator 150 is adjustable by means of flexible and stretchablesilicone and/or rubber strip 154 and a loop-closing means 156. Theloop-closing means 156 is adapted to close a loop with the flexible andstretchable strip. Together, the flexible and stretchable strip and theloop-closing means are provided so as to fit the wearable operator to atleast one selected from a group consisting of (a) said predeterminedlocation of said different instruments; (b) said predetermined body partof said surgeon, each of which is characterized by a different size andshape.

As will be disclosed hereinafter, the loop-closing means 156 can bee.g., a unidirectional catch, a rack, a peg or any other mechanism knownin the art.

According to some embodiments, the silicone and/or rubber strip 154 ispassed through a unidirectional catch (e.g., ratchet 156), such that,when the physician wears the wearable operator 150, he adjusts the sameby pulling the silicone and/or rubber strip 154 through the ratchet 156.

According to some embodiments, the silicone and/or rubber strip 154 isrotated around rack or peg 156 such that, when the physician wears thewearable operator 150, he adjusts the same by pulling the siliconeand/or rubber strip 154 around the peg 156.

According to this embodiment, the silicone and/or rubber strip 154 ischaracterized by a varied width along its length. More specifically, atleast a portion of the silicone and/or rubber strip 154 is characterizedby a greater width, such that when the same is twisted/rotated aroundpeg 156 and reaches the wider portion, the same is fixedly secured tothe wearable operator 150.

According to some embodiments, the silicone and/or rubber strip 154 ischaracterized by different surface roughnesses along its length. Morespecifically, at least a portion of the silicone and/or rubber strip 154is characterized by e.g., an abrasive or rough surface such that whenthe same is twisted/rotated around peg 156 and reaches the rougherportion, the same is fixedly secured to the wearable operator 150.

Referring again to FIG. 27 c illustrating an embodiment of the wearableoperator 150 attached to a surgical tool via fastener 155. Someembodiments of fastener 155 are shown in FIGS. 27 e -27 i.

According to the embodiment shown in FIG. 27 e , the wearable operator150 comprises a unidirectional coupling (e.g., ratchet 1510).

Once the wearable operator 150 is secured to the surgical tool, thewearable operator 150 is adjusted to the size and dimensions of thesurgical tool by means of a unidirectional catch (e.g., ratchet 1510).

According to some embodiments, the wearable operator 150 comprises abody having at least two portions 1520 and 1521 (see FIG. 27 f ). Saidportions are adapted to ‘grasp’ the surgical tool such that when thewearable operator 150 is coupled to the surgical tool, fine-tunedmovement of the two body portions is obtainable so as to provide saidtight-fit coupling between said two portions and said instrument.

According to some embodiments (FIG. 27 g ), one of the two portions(either 1520 or 1521) is rotationally movable relative to the other,such that when said wearable operator is coupled to said instrument,fine-tuned movement of said two body portions is obtainable so as toprovide said tight-fit coupling between said two portions and saidinstrument.

According to some embodiments (FIG. 27 h ), the two portions (1521 and1520) are rotationally movable relative to each other, such that whenthe wearable operator is coupled to said instrument, fine-tuned movementof said two body portions is obtainable so as to provide said tight-fitcoupling between said two portions and said instrument.

In reference to FIG. 27 h , the movement of either portion 1520 orportion 1521 relative to the other is obtained by fixating the positionof either portion 1520 or portion 1521 and coupling the other portion toe.g., a unidirectional catch (e.g., ratchet) 1510 or a two-waydirectional catch 1510 on the body of the wearable operator.

According to some embodiments, the movement of either portion 1520 orportion 1521 relative to the other is obtained by providing one portion,e.g., portion 1521 with cog-like teeth 1511 and the body of the wearableoperator with cog-like teeth 1512 matching with cog-like teeth 1511 (seeFIG. 27 i ). In such a way portion 1521 can be linearly moved relativeto portion 1520.

Referring again to FIG. 27 d , this embodiment of operator 150 depictsjoystick unit 152 a, which controls maneuvering of the endoscope,comprising a force joystick, while joystick unit 152 b, which controlszooming, comprises a pressable button. Button 156 is an emergency offbutton; pressing button 156 quickly stops all motion. Light 158 is afault light, illuminating when a fault is detected in the system.

In some embodiments, a single control is used for both lateralmaneuvering of the endoscope and zooming of the endoscope. In somevariants of these embodiments, lateral movement of the joystick unit orlateral movement of a hand on the joystick unit is translated intolateral maneuvering of the endoscope, for non-limiting example, asdescribed above for FIG. 26 b , while pressure on the joystick unit,either continuous pressure or clicking the joystick unit, is translatedinto zooming of the endoscope, as described hereinbelow.

In some embodiments, two joystick units are used, one for lateralmaneuvering of the endoscope, and one for zooming.

In some embodiments, two joystick units are used, one for maneuvering,both lateral maneuvering and zoom, and the other for directing theendoscope to focus on a desired tool. In these embodiments, on thedisplay screen showing the field of view of the endoscope, a symbolindicates the tool on which the endoscope is focused. This symbol can bea shape or it can be highlighting. When a user clicks on the secondjoystick unit, the new tool to be focused on is indicated, either bymoving the symbol or highlighting to the new tool, or by a second symbolor a second color of highlighting. The user can repeat clicking untilthe desired tool is indicated. In some embodiments, ceasing to click onthe second joystick unit indicates that the current tool is the desiredtool; in some embodiments, a longer pressure on the second joystick unitindicates that the current tool is the desired tool. Once the desiredtoo has been selected, the endoscope redirects to the desired tool.

Operation of the zoom mechanism can be by clicking on a joystick unit orby a continuous pressure on a joystick unit. Some non-limiting examplesof embodiments of methods of zoom control include:

-   -   1. A single click to select zoom in, a double click to select        zoom out, and continuous pressure to zoom at a predetermined        rate in the selected direction.    -   2. A double click to select zoom in, a single click to select        zoom out, and continuous pressure to zoom in the selected        direction.    -   3. A single click to change the direction of zoom and continuous        pressure to zoom at a predetermined rate in the selected        direction.    -   4. A single click to change the direction of the zoom, a double        click to zoom by a predetermined amount in the selected        direction.    -   5. A double click to change the direction of the zoom, a single        click to zoom by a predetermined amount in the selected        direction.    -   6. A single click to change the direction of zoom and a double        click to zoom by a predetermined amount in the selected        direction.    -   7. A single click to change the direction of zoom, a double        click to zoom by a predetermined amount in the selected        direction, and continuous pressure to zoom at a predetermined        rate in the selected direction.

In embodiments in which continuous pressure is used to zoom in theselected direction, in some variants of these embodiments, continuouspressure above a predetermined minimum pressure (a minimum which can bezero) zooms the endoscope at a predetermined zoom rate. In othervariants of embodiments in which continuous pressure is used to zoom inthe selected direction, if the pressure is above a predetermined minimumpressure, which can be zero, the greater the pressure, the greater thezoom rate, until a predetermined maximum rate is reached, above whichthe zoom rate is the predetermined maximum rate.

In embodiments wearable by the user, the operator 150 can be worn as aring on a finger; as a wristband on the wrist; an armband on an arm; onthe chest, either supported around the chest or supported around theneck; or on the head, supported by a headband, by a helmet or by ahelmet frame.

The communication means connecting the maneuvering system to theoperator 150 can be a wireless communication means, a wiredcommunication means, and any combination thereof.

In some embodiments of the current invention, SFME 130 additionallycomprises, in a non-limiting manner, means for controlling movement ofendoscope 200 adapted to restrain the endoscope's velocity.

In some embodiments of the current invention, SFME 130 additionallycomprises, in a non-limiting manner, n sensors, where n is an integergreater than or equal to one. The sensors may be adapted to activate incase of power failure or to activate when connected to power. Thesensors are selected in a non-limiting manner from a group consisting,for example, of motion sensors, heat sensors, electric sensors, soundsensors, pressure sensors, optical sensors, and any combination thereof.

In some embodiments of the current invention, joystick unit 152 ischaracterized in a non-limiting manner by an external surface.

In some embodiments of the current invention, at least one motion sensordetects motion of joystick unit 152. Furthermore, detection of motion isused for deactivation of the motion of endoscope 200 if the requestedspeed of the motion is above a predetermined threshold.

In some embodiments of the current invention, at least one motion sensordetects, in a non-limiting manner, motion on the external surface ofjoystick unit 152. Furthermore, endoscope 200 then moves in response tothe motion on the external surface of joystick unit 152. Additionally,detection of motion above a predetermined threshold speed on joystickunit 152 will deactivate motion of endoscope 200.

In some embodiments of the current invention, at least one heat sensoris adapted in a non-limiting manner to sense temperatures in the rangeof about 35 to about 42 degrees. The at least one heat sensor is adaptedto sense whether a human hand/fingers are activating (i.e., touching)the joystick unit 152.

Furthermore, at least one heat sensor enables in a non-limiting mannerthe activation of SFME 130 when the at least one heat sensor sensestemperatures in the range of about 35 to about 42 degrees.

Additionally, at least one heat sensor is adapted in a non-limitingmanner to provide a thermal image, where the at least one heat sensor iscoupled to a processing unit adapted to provide the endoscope user withthe thermal image, and a processing unit enables the activation of SFME130 upon analysis of the image and detection of human hand.

In some embodiments of the current invention, at least one electricsensor is adapted in a non-limiting manner to detect, for example, apower failure, the electric conductivity of the subject's body, and anycombination thereof. Additionally, the conductivity of the subject'sbody sensed by the at least one electric sensor enables the activationof the SFME.

In some embodiments of the current invention, at least one sound sensoris adapted in a non-limiting manner to sense predetermined soundpatterns. Furthermore, the predetermined sound patterns sensed by the atleast one sound sensor enables the activation of SFME 130. Additionally,at least one sound sensor is used to operate endoscope 200 according topredetermined sound patterns (e.g., the human voice, predeterminedmovement commands).

In some embodiments of the current invention, at least one pressuresensor is adapted in a non-limiting manner to sense pressure applied toSFME 130.

Additionally, in some embodiments, the pressure sensed by the at leastone pressure sensor is used to activate SFME 130. In some embodiments,when the pressure sensed by the at least one pressure sensor is above apredetermined threshold, SFME 130 is activated. In some embodiments,when the pressure sensed by the at least one pressure sensor is below apredetermined threshold, SFME 130 is de-activated. In some embodiments,when the pressure sensed by the at least one pressure sensor is below apredetermined threshold, SFME 130 is activated.

An example of pressure above a pre-determined threshold activating SFME130 is a pressure sensor in the joystick unit, which is activated whenthe pressure of a hand or fingers or an appropriated implement issensed. SFME 130 would be deactivated if the above pressure was below apre-determined threshold.

An example of pressure above a pre-determined threshold de-activatingSFME 130 is a joystick unit where pressure is used to set the speed ofmotion of a scalpel. If the pressure is above the pre-determinedthreshold, the scalpel would move too rapidly, so SFME 130 isde-activated if pressures above the pre-determined threshold are sensed.SFME 130 would then be activated if pressures are below a pre-determinedthreshold.

In some embodiments of the current invention, at least one opticalsensor is adapted in a non-limiting manner to sense visual changesaccording to predetermined visual patterns. Furthermore, the at leastone optical sensor enables the activation of SFME 130 according topredetermined visual patterns. Additionally, at least one optical sensoris used to operate endoscope 200 according to predetermined visualpatterns.

In some embodiments, SFME 130 is adapted to sense power failure by anymeans known in the art, including the sensors described herein. In someembodiments, SFME 130 responds to power failure by instructing themaneuvering system to keep the endoscope and any other controlledinstruments in the position and at the angle held by them immediatelybefore the power failure. In some embodiments, the system furthercomprises means by which the endoscope can be manually switched tomanual control in the event of power failure, so that the operation cancontinue safely with an operating assistant maneuvering the endoscopeduring the period of power failure.

In some embodiments, the system comprises battery backup such that, inthe event of power failure, the system switches automatically to batterypower, enabling the SFME to continue to control movement of theendoscope during power outages.

In some embodiments, a standard articulating endoscope, such as theStryker™ articulating endoscope is used. In some embodiments, anintegral articulating endoscope is used.

According to some embodiments of the present invention, the surgicalcontrolling system comprises the following components:

-   -   a. at least one surgical tool adapted to be inserted into a        surgical environment of a human body for assisting a surgical        procedure, at least one said tool being an articulating tool;    -   b. at least one location estimating means adapted to real-time        estimate/locate the location (i.e., the 3D spatial position) of        the at least one surgical tool at any given time t;    -   c. at least one movement detection means communicable with a        movement-database and with said location estimating means; said        movement-database is adapted to store said 3D spatial position        of said at least one surgical tool at time t_(f) and at time t₀,        where t_(f)>t₀; said movement detection means is adapted to        detect movement of said at least one surgical tool if the 3D        spatial position of said at least one surgical tool at time        t_(f) is different than said 3D spatial position of said at        least one surgical tool at time t₀, and,    -   d. a controller having a processing means communicable with a        database, the controller adapted to control the spatial position        of the at least one surgical tool.

The initial time t₀ can be the beginning of the surgical procedure, itcan be the time at which the tool entered the body, it can be the timeat the beginning of the current movement, or it can be the previoustimestep in the current maneuver. In preferred embodiments, theprocessor will reset t₀ as necessary during the surgical procedure. Fornon-limiting example, the difference in position between the location ofthe tool at the previous timestep and its location at the currenttimestep can be used to calculate the tool's current velocity while thedifference in position between its current position and its position atthe start of the current maneuver can be used to calculate the tool'soverall direction of motion.

The location of the tool can be the location of the tool's tip, thelocation of a predetermined point on the tool's body, or the location ofa predetermined point on the tool's handle. The position defining thelocation of the tool can be changed as needed, e.g., from the locationof the body to the location of the tip.

In some embodiments, the surgical controlling system additionallycomprises a touchscreen adapted to accept input of a location within thebody, that location indicated by pressure on the portion of thetouchscreen showing the image of the location.

In order to facilitate control, a number of motion control rules havebeen implemented, as described hereinbelow.

It is within the scope of the present invention that the database isadapted to store a predetermined set of rules according to which ALLOWEDand RESTRICTED movements of the at least one surgical tool aredetermined, such that the spatial position of the at least one surgicaltool is controlled by the controller according to the ALLOWED andRESTRICTED movements. In other words, each detected movement by saidmovement detection means of said at least one surgical tool isdetermined as either an ALLOWED movement or as a RESTRICTED movementaccording to said predetermined set of rules.

Thus, the present invention stores the 3D spatial position of eachsurgical tool at a current at time t_(f) and at time to; where t_(f)>t₀.If the 3D spatial position of said at least one surgical tool at timet_(f) is different than said 3D spatial position of said at least onesurgical tool at time to movement of the tool is detected. Next thesystem analyses said movement according to said set of rule and processwhether said movement is ALLOWED movement or RESTRICTED movement.

According to one embodiment of the present invention, the systemprevents said movement, if said movement is a RESTRICTED movement. Saidmovement prevention is obtained by controlling a maneuvering systemwhich prevents the movement of said surgical tool.

According to one embodiment of the present invention, the system doesnot prevent said movement, (if said movement is a RESTRICTED movement),but merely signals/alerts the user (i.e., the physician) of saidRESTRICTED movement.

According to some embodiments of the present invention, said surgicaltool is an endoscope.

According to some embodiments of the present invention, the controllermay provide a suggestion to the operator as to which direction thesurgical tool has to move to or may be moved to.

Thus, according to a preferred embodiment of the present invention, thepresent invention provides a predetermined set of rules which definewhat is an “ALLOWED movement” of any surgical tool within the surgicalenvironment and what is a “RESTRICTED movement” of any surgical toolwithin the surgical environment.

According to some embodiments the system of the present inventioncomprises a maneuvering subsystem communicable with the controller, themaneuvering subsystem is adapted to spatially reposition the at leastone surgical tool during surgery according to the predetermined set ofrules.

According to some embodiments, the controller may provide instructionsto a maneuvering subsystem for spatially repositioning the location ofthe surgical tool. According to these instructions, only ALLOWEDmovements of the surgical tool will be performed. Preventing RESTRICTEDmovements is performed by: detecting the location of the surgical tool;processing all current rules; analyzing the movement of the surgicaltool and preventing the movement if the tool's movement is a RESTRICTEDmovement.

According to some embodiments, system merely alerts the physician of aRESTRICTED movement of at least one surgical tool (instead of preventingsaid RESTRICTED movement).

Alerting the physician of RESTRICTED movements (or, alternativelypreventing a RESTRICTED movement) is performed by: detecting thelocation of the surgical tool; processing all current rules; analyzingthe movement of the surgical tool and informing the surgeon (the user ofthe system) if the tool's movement is an ALLOWED movement or aRESTRICTED movement.

Thus, according to a preferred embodiment of the present invention, ifRESTRICTED movements are prevented, the same process (of detecting thelocation of the surgical tool; processing all current rules andanalyzing the movement of the surgical tool) is followed except for thelast movement, where the movement is prevented if the tool's movement isa RESTRICTED movement. The surgeon can also be informed that themovement is being prevented.

According to some embodiment, the above (alerting the physician and/orpreventing the movement) is performed by detecting the location of thesurgical tool and analyzing the surgical environment of the surgicaltool. Following analysis of the surgical environment and detection ofthe location of the surgical tool, the system may assess all the riskswhich may follow a movement of the surgical tool in the predetermineddirection. Therefore, each location in the surgical environment has tobe analyzed so that any possible movement of the surgical tool will beclassified as an ALLOWED movement or a RESTRICTED movement.

According to one embodiment of the present invention, the location ofeach tool is determined using image processing means and determining inreal-time what is the 3D spatial location of each tool. It should beunderstood that the above mentioned “tool” may refer to the any locationon the tool. For example, it can refer to the tip of the same, the bodyof the same and any combination thereof.

In some embodiments, avoidance of body organs is facilitated by means ofa proximity sensor on the circumference of at least one tool. In theseembodiments, if the distance between the tool and another object in thesurgical environment, such as, but not limited to, an organ or anothertool, is less than a predetermined distance, the proximity sensoractivates, thereby notifying the control system that at least one toolis too close to another object in the surgical environment.

In some variants of embodiments with proximity sensors, the proximitysensor not only determined whether an object is within a predetermineddistance of the sensor, it also determines, for objects within thepredetermined distance, the distance between the sensor and the object.

Hereinbelow, determination of the 3D location of each tool includesdetermination by means of a proximity sensor as well as determination bymeans of image processing.

The predetermined set of rules which are the essence of the presentinvention are adapted to take into consideration all the possiblefactors which may be important during the surgical procedure. Thepredetermined set of rules may comprise the following rules or anycombination thereof:

-   -   a. a route rule;    -   b. an environment rule;    -   c. an operator input rule;    -   d. a proximity rule;    -   e. a collision prevention rule;    -   f. a history based rule;    -   g. a tool-dependent ALLOWED and RESTRICTED movements rule.    -   h. a most used tool rule;    -   i. a right tool rule;    -   j. a left tool rule;    -   k. a field of view rule;    -   l. a no fly zone rule;    -   m. a preferred volume zone rule;    -   n. a preferred tool rule;    -   o. a movement detection rule, and    -   p. a tagged tool rule.

Thus, for example, the collision prevention rule defines a minimumdistance below which two or more tools should not be brought together(i.e., there is minimum distance between two or more tools that shouldbe maintained). If the movement of one tool will cause it to comedangerously close to another tool (i.e., the distance between them,after the movement, is smaller than the minimum distance defined by thecollision prevention rule), the controller either alerts the user thatthe movement is a RESTRICTED movement or does not permit the movement.

It should be emphasized that all of the above (and the followingdisclosure) is enabled by constantly monitoring the surgicalenvironment, and identifying and locating the 3D spatial location ofeach element/tool in the surgical environment.

The identification is provided by conventional means known to anyskilled in the art (e.g., image processing, optical means etc.).

The following provides explanations for each of the above mentionedrules and its functions:

According to some embodiments, the route rule comprises a predefinedroute in which the at least one surgical tool is adapted to move withinthe surgical environment; the ALLOWED movements are movements in whichthe at least one surgical tool is located within the borders of thepredefined route, and the RESTRICTED movements are movements in whichthe at least one surgical tool is located out of the borders of thepredefined route. Thus, according to this embodiment, the route rulecomprises a communicable database storing at least one predefined routein which the at least one surgical tool is adapted to move within thesurgical environment; the predefined route comprises n 3D spatialpositions of the at least one surgical tool in the route; n is aninteger greater than or equal to 2; ALLOWED movements are movements inwhich the at least one surgical tool is located substantially in atleast one of the n 3D spatial positions of the predefined route, andRESTRICTED movements are movements in which the location of the at leastone surgical tool is substantially different from the n 3D spatialpositions of the predefined route.

In other words, according to the route rule, each of the surgical tool'scourses (and path in any surgical procedure) is stored in a communicabledatabase. ALLOWED movements are defined as movements in which the atleast one surgical tool is located substantially in at least one of thestored routes; and RESTRICTED movements are movements in which the atleast one surgical tool is in a substantially different location thanany location in any stored route.

According to some embodiments, the environmental rule is adapted todetermine ALLOWED and RESTRICTED movements according to hazards orobstacles in the surgical environment as received from an endoscope orother sensing means. Thus, according to this embodiment, theenvironmental rule comprises a comprises a communicable database; thecommunicable database is adapted to received real-time images of thesurgical environment and is adapted to perform real-time imageprocessing of the same and to determine the 3D spatial position ofhazards or obstacles in the surgical environment; the environmental ruleis adapted to determine ALLOWED and RESTRICTED movements according tohazards or obstacles in the surgical environment, such that RESTRICTEDmovements are movements in which at least one surgical tool is locatedsubstantially in at least one of the 3D spatial positions, and ALLOWEDmovements are movements in which the location of at least one surgicaltool is substantially different from the 3D spatial positions.

In other words, according to the environment rule, each element in thesurgical environment is identified so as to establish which is a hazardor obstacle (and a path in any surgical procedure) and each hazard andobstacle (and path) is stored in a communicable database. RESTRICTEDmovements are defined as movements in which the at least one surgicaltool is located substantially in the same location as that of thehazards or obstacles; and the ALLOWED movements are movements in whichthe location of the at least one surgical tool is substantiallydifferent from that of all of the hazards or obstacles.

According to some embodiments, hazards and obstacles in the surgicalenvironment are selected from a group consisting of tissues, surgicaltools, organs, endoscopes and any combination thereof.

According to some embodiments, the operator input rule is adapted toreceive an input from the operator of the system regarding the ALLOWEDand RESTRICTED movements of the at least one surgical tool. In otherwords, the operator input rule receives instructions from the operatorof the system as to what can be regarded as ALLOWED movements and whatare RESTRICTED movements.

According to some embodiments, the input comprises n 3D spatialpositions; n is an integer greater than or equal to 2; wherein at leastone of which is defined as an ALLOWED location and at least one of whichis defined as a RESTRICTED location, such that the ALLOWED movements aremovements in which the at least one surgical tool is locatedsubstantially in at least one of the n 3D ALLOWED spatial positions, andthe RESTRICTED movements are movements in which the location of the atleast one surgical tool is substantially different from the n 3D ALLOWEDspatial positions.

According to some embodiments, the input comprises at least one ruleaccording to which ALLOWED and RESTRICTED movements of the at least onesurgical tool are determined, such that the spatial position of the atleast one surgical tool is controlled by the controller according to theALLOWED and RESTRICTED movements.

According to some embodiments, the operator input rule can convert anALLOWED movement to a RESTRICTED movement and a RESTRICTED movement toan ALLOWED movement.

According to some embodiments, the proximity rule is adapted to define apredetermined distance between the at least one surgical tool and atleast one another surgical tool; the ALLOWED movements are movementswhich are within the range or out of the range of the predetermineddistance, and the RESTRICTED movements which are out of the range orwithin the range of the predetermined distance; the ALLOWED movementsand the RESTRICTED movements are defined according to different ranges.Thus, according to this embodiment, the proximity rule is adapted todefine a predetermined distance between at least two surgical tools. Ina preferred embodiment, the ALLOWED movements are movements which arewithin the range of the predetermined distance, while the RESTRICTEDmovements which are out of the range of the predetermined distance. Inanother preferred embodiment, the ALLOWED movements are movements whichare out of the range of the predetermined distance, while the RESTRICTEDmovements are within the range of the predetermined distance

It should be pointed out that the above mentioned distance can beselected from the following:

-   -   (a) the distance between the tip of the first tool and the tip        of the second tool;    -   (b) the distance between the body of the first tool and the tip        of the second tool;    -   (c) the distance between the body of the first tool and the body        of the second tool; and    -   (d) the distance between the tip of the first tool and the body        of the second tool; and any combination thereof.

According to some embodiments, the proximity rule is adapted to define apredetermined angle between at least three surgical tools; ALLOWEDmovements are movements which are within the range or out of the rangeof the predetermined angle, and RESTRICTED movements are movements whichare out of the range or within the range of the predetermined angle.

According to some embodiments, the collision prevention rule is adaptedto define a predetermined distance between the at least one surgicaltool and an anatomical element within the surgical environment (e.g.tissue, organ, another surgical tool or any combination thereof); theALLOWED movements are movements which are in a range that is larger thanthe predetermined distance, and the RESTRICTED movements are movementswhich is in a range that is smaller than the predetermined distance.

According to some embodiments, the anatomical element is selected from agroup consisting of tissue, organ, another surgical tool or anycombination thereof.

According to some embodiments, the surgical tool is an endoscope. Theendoscope is adapted to provide real-time images of the surgicalenvironment.

According to some embodiments, the right tool rule is adapted todetermine the ALLOWED movement of the endoscope according to themovement of a surgical tool in a specified position in relation to theendoscope, preferably positioned to right of the same. According to thisrule, the tool which is defined as the right tool is constantly trackedby the endoscope. According to some embodiments, the right tool isdefined as the tool positioned to the right of the endoscope; accordingto some embodiments, any tool can be defined as the right tool. AnALLOWED movement, according to the right tool rule, is a movement inwhich the endoscope field of view is moved to a location substantiallythe same as the location of the right tool, thereby tracking the righttool. A RESTRICTED movement, according to the right tool rule, is amovement in which the endoscope field of view is moved to a locationsubstantially different from the location of the right tool.

According to some embodiments, the left tool rule is adapted todetermine the ALLOWED movement of the endoscope according to themovement of a surgical tool in a specified position in relation to theendoscope, preferably positioned to left of the same. According to thisrule, the tool which is defined as the left tool is constantly trackedby the endoscope. According to some embodiments, the left tool isdefined as the tool positioned to the left of the endoscope; accordingto some embodiments, any tool can be defined as the left tool. AnALLOWED movement, according to the left tool rule, is a movement inwhich the endoscope field of view is moved to a location substantiallythe same as the location of the left tool. A RESTRICTED movement,according to the left tool rule, is a movement in which the endoscopefield of view is moved to a location substantially different from thelocation of the left tool.

According to some embodiments, the field of view rule is adapted todefine a field of view and maintain that field of view. The field ofview rule is defined such that if the endoscope is adapted to track apredetermined set of tools in a desired field of view, when one of thosetools is no longer in the field of view, the rule instructs theendoscope to zoom out so as to reintroduce the tool into the field ofview. Thus, according to this embodiment, the field of view rulecomprises a communicable database comprising n 3D spatial positions; nis an integer greater than or equal to 2; the combination of all of then 3D spatial positions provides a predetermined field of view; the fieldof view rule is adapted to determine the ALLOWED movement of theendoscope within the n 3D spatial positions so as to maintain a constantfield of view, such that the ALLOWED movements are movements in whichthe endoscope is located substantially in at least one of the n 3Dspatial positions, and the RESTRICTED movements are movements in whichthe location of the endoscope is substantially different from the n 3Dspatial positions.

Thus, according to some embodiments of the field of view rule, the fieldof view rule comprises a communicable database comprising n 3D spatialpositions; n is an integer greater than or equal to 2; the combinationof all of the n 3D spatial positions provides a predetermined field ofview. The field of view rule further comprises a communicable databaseof m tools and the 3D spatial locations of the same, where m is aninteger greater than or equal to 1 and where a tool can be a surgicaltool, an anatomical element and any combination thereof. The combinationof all of the n 3D spatial positions provides a predetermined field ofview. The field of view rule is adapted to determine ALLOWED movement ofthe endoscope such that the m 3D spatial positions of the tools compriseat least one of the n 3D spatial positions of the field of view, andRESTRICTED movements are movements in which the 3D spatial position ofat least one tool is substantially different from the n 3D spatialpositions of the field of view.

According to some embodiments, the preferred volume zone rule comprisesa communicable database comprising n 3D spatial positions; n is aninteger greater than or equal to 2; the n 3D spatial positions providesthe preferred volume zone; the preferred volume zone rule is adapted todetermine the ALLOWED movement of the endoscope within the n 3D spatialpositions and RESTRICTED movement of the endoscope outside the n 3Dspatial positions, such that the ALLOWED movements are movements inwhich the endoscope is located substantially in at least one of the n 3Dspatial positions, and the RESTRICTED movements are movements in whichthe location of the endoscope is substantially different from the n 3Dspatial positions. In other words, the preferred volume zone ruledefines a volume of interest (a desired volume of interest), such thatan ALLOWED movement, according to the preferred volume zone rule, is amovement in which the endoscope (or any surgical tool) is moved to alocation within the defined preferred volume. A RESTRICTED movement,according to the preferred volume zone rule, is a movement in which theendoscope (or any surgical tool) is moved to a location outside thedefined preferred volume.

According to some embodiments, the preferred tool rule comprises acommunicable database, the database stores a preferred tool; thepreferred tool rule is adapted to determine the ALLOWED movement of theendoscope according to the movement of the preferred tool. In otherwords, the preferred tool rule defines a preferred tool (i.e., a tool ofinterest) that the user of the system wishes to track. An ALLOWEDmovement, according to the preferred tool rule, is a movement in whichthe endoscope is moved to a location substantially the same as thelocation of the preferred tool. A RESTRICTED movement is a movement inwhich the endoscope is moved to a location substantially different fromthe location of the preferred tool. Thus, according to the preferredtool rule the endoscope constantly tracks the preferred tool, such thatthe field of view, as seen from the endoscope, is constantly thepreferred tool. It should be noted that the user may define in saidpreferred tool rule to constantly track the tip of said preferred toolor alternatively, the user may define in said preferred tool rule toconstantly track the body or any location on the preferred tool.

According to some embodiments, the no fly zone rule is adapted to definea RESTRICTED zone into which no tool (or alternatively no predefinedtool) is permitted to enter. Thus, according to this embodiment, the nofly zone rule comprises a communicable database comprising n 3D spatialpositions; n is an integer greater than or equal to 2; the n 3D spatialpositions define a predetermined volume within the surgical environment;the no fly zone rule is adapted to determine a RESTRICTED movement ifthe movement is within the no fly zone and an ALLOWED movement if themovement is outside the no fly zone, such that RESTRICTED movements aremovements in which the at least one surgical tool is locatedsubstantially in at least one of the n 3D spatial positions, and theALLOWED movements are movements in which the location of the at leastone surgical tool is substantially different from the n 3D spatialpositions.

According to some embodiments, the most used tool rule is adapted todefine (either real-time, during the procedure or prior to theprocedure) which tool is the most used tool (i.e., the tool which ismoved the most during the procedure) and to instruct the maneuveringsubsystem to constantly position the endoscope to track the movement ofthis tool. Thus, according to this embodiment, the most used tool rulecomprises a communicable database counting the number of movements ofeach of the surgical tools; the most used tool rule is adapted toconstantly position the endoscope to track the movement of the surgicaltool with the largest number of movements. In some embodiments of themost used tool rule, the communicable database measures the amount ofmovement of each of the surgical tools; the most used tool rule isadapted to constantly position the endoscope to track the movement ofthe surgical tool with the largest amount of movement.

According to some embodiments, the system is adapted to alert thephysician of a RESTRICTED movement of at least one surgical tool. Thealert can be audio signaling, voice signaling, light signaling, flashingsignaling and any combination thereof.

According to some embodiments, an ALLOWED movement is one permitted bythe controller and a RESTRICTED movement is one denied by thecontroller.

According to some embodiments, the history-based rule is adapted todetermine the ALLOWED and RESTRICTED movements according to historicalmovements of the at least one surgical tool in at least one previoussurgery. Thus, according to this embodiment, the history-based rulecomprises a communicable database storing each 3D spatial position ofeach of the surgical tools, such that each movement of each surgicaltool is stored; the history-based rule is adapted to determine ALLOWEDand RESTRICTED movements according to historical movements of the atleast one surgical tool, such that the ALLOWED movements are movementsin which the at least one surgical tool is located substantially in atleast one of the 3D spatial positions, and the RESTRICTED movements aremovements in which the location of the at least one surgical tool issubstantially different from the n 3D spatial positions.

According to some embodiments, the tool-dependent ALLOWED and RESTRICTEDmovements rule is adapted to determine ALLOWED and RESTRICTED movementsaccording to predetermined characteristics of the surgical tool, wherethe predetermined characteristics of the surgical tool are selected froma group consisting of: physical dimensions, structure, weight,sharpness, and any combination thereof. Thus, according to thisembodiment, the tool-dependent ALLOWED and RESTRICTED movements rulecomprises a communicable database; the communicable database is adaptedto store predetermined characteristics of at least one of the surgicaltools; the tool-dependent ALLOWED and RESTRICTED movements rule isadapted to determine ALLOWED and RESTRICTED movements according to thepredetermined characteristics of the surgical tool.

According to these embodiments, the user can define, e.g., the structureof the surgical tool he wishes the endoscope to track. Thus, accordingto the tool-dependent ALLOWED and RESTRICTED movements rule theendoscope constantly tracks the surgical tool having predeterminedcharacteristics as defined by the user.

According to some embodiments of the present invention, the movementdetection rule comprises a communicable database comprising thereal-time 3D spatial positions of each surgical tool; said movementdetection rule is adapted to detect movement of at least one surgicaltool. When a change in the 3D spatial position of that surgical tool isreceived, ALLOWED movements are movements in which the endoscope isre-directed to focus on the moving surgical tool.

According to some embodiments of the present invention, the tagged toolrule comprises means of tagging at least one surgical tool within thesurgical environment such that, by maneuvering the endoscope, theendoscope is constantly directed to the tagged surgical tool. Thus,according to the tagged tool rule, the endoscope constantly tracks thepreferred (i.e., tagged) tool, such that the field of view, as seen fromthe endoscope, is constantly maintained on the preferred (tagged) tool.It should be noted that the user can define the tagged tool rule toconstantly track the tip of the preferred (tagged) tool, the body of thepreferred (tagged) tool, or any other location on the preferred (tagged)tool.

According to some embodiments of the present invention, the systemfurther comprises a maneuvering subsystem communicable with thecontroller. The maneuvering subsystem is adapted to spatially repositionthe at least one surgical tool during a surgery according to thepredetermined set of rules.

According to some embodiments, the at least one location estimatingmeans is at least one endoscope adapted to acquire real-time images of asurgical environment within the human body for the estimation of thelocation of at least one surgical tool.

According to some embodiments, the location estimating means comprise atleast one selected from a group consisting of optical imaging means,radio frequency transmitting and receiving means, at least one mark onat least one surgical tool and any combination thereof.

According to some embodiments, the at least one location estimatingmeans is an interface subsystem between a surgeon and at least onesurgical tool, the interface subsystem comprising (a) at least one arraycomprising N regular light sources or N pattern light sources, where Nis a positive integer; (b) at least one array comprising M cameras,where M is a positive integer; (c) optional optical markers and meansfor attaching the optical markers to at least one surgical tool; and (d)a computerized algorithm operable via the controller, the computerizedalgorithm adapted to record images received by each camera of each ofthe M cameras and to calculate therefrom the position of each of thetools, and further adapted to provide automatically the results of thecalculation to the human operator of the interface.

It is well known that surgery is a highly dynamic procedure with aconstantly changing environment which depends on many variables. Anon-limiting list of these variables includes, for example: the type ofthe surgery, the working space (e.g., with foreign objects, dynamicuncorrelated movements, etc), the type of tools used during the surgery,changing background, relative movements, dynamic procedures, dynamicinput from the operator and the history of the patient. Therefore, thereis need for a system which is able to integrate all the variables byweighting their importance and deciding to which spatial position theendoscope should be relocated.

The present invention can be also utilized to improve the interfacebetween the operators (e.g., the surgeon, the operating medicalassistant, the surgeon's colleagues, etc.). Moreover, the presentinvention can be also utilized to control and/or direct an automatedmaneuvering subsystem to focus the endoscope on an instrument selectedby the surgeon, or to any other region of interest. This may beperformed in order to estimate the location of at least one surgicaltool during a surgical procedure.

The present invention also discloses a surgical tracking system which isadapted to guide and relocate an endoscope to a predetermined region ofinterest in an automatic and/or a semi-automatic manner. This operationis assisted by an image processing algorithm(s) which is adapted toanalyze the received data from the endoscope in real time, and to assessthe surgical environment of the endoscope.

According to an embodiment, the system comprises a “smart” trackingsubsystem, which receives instructions from a maneuvering function f(t)(t is the time) as to where to direct the endoscope and which instructsthe maneuvering subsystem to relocate the endoscope to the requiredarea.

The maneuvering function f(t) receives, as input, output from at leasttwo instructing functions g_(i)(t), analyses their output and providesinstruction to the “smart” tracking system (which eventually re-directsthe endoscope).

The instructing functions g_(i)(t) of the present invention arefunctions which are configured to assess the environment of theendoscope and the surgery, and to output data which guides the trackingsubsystem for controlling the spatial position of the maneuveringsubsystem and the endoscope. The instructing functions g_(i)(t) may beselected from a group consisting of:

-   -   a. a tool detection function g₁(t);    -   b. a movement detection function g₂(t);    -   c. an organ detection function g₃(t);    -   d. a collision detection function g₄(t);    -   e. an operator input function g₅(t);    -   f. a prediction function g₆(t);    -   g. a past statistical analysis function g₇(t);    -   h. a most used tool function g₈(t);    -   i. a right tool function g₉(t);    -   j. a left tool function g₁₀(t);    -   k. a field of view function g₁₁(t);    -   l. a preferred volume zone function g₁₂(t);    -   m. a no fly zone function g₁₃(t);    -   n. a proximity function g₁₄(t);    -   o. a tagged tool function g₁₅(t);    -   p. a preferred tool function g₁₆(t).

According to some embodiments, each instructing function g_(i)(t) isalso given a weighting function, α_(i)(t). The purpose of the weightingfunction is to enable the system to select the instructing function tobe executed, if more than one instructing function is activatable at onetime.

Thus, for non-limiting example, the maneuvering function f(t) receivesinput from two instructing functions: the collision detection functiong₄(t) (the function providing information whether the distance betweentwo elements is smaller than a predetermined distance) and from the mostused tool function g₈(t) (the function counts the number of times eachtool is moved during a surgical procedure and provides information as towhether the most moved or most used tool is currently moving). Theoutput given from the collision detection function g₄(t) is that asurgical tool is dangerously close to an organ in the surgicalenvironment. The output given from the most used tool function g₈(t) isthat the tool identified statistically as the most moved tool iscurrently moving.

The maneuvering function f(t) then assigns each of the instructingfunctions with weighting functions α_(i)(t). For the above example, themost used tool function g₈(t) is assigned with a greater weight than theweight assigned to the collision detection function g₄(t).

After the maneuvering function f(t) analyses the information receivedfrom the instructing functions g_(i)(t) and the weighting functionsα_(i)(t) of each, the maneuvering function f(t) outputs instructions tothe maneuvering subsystem to re-direct the endoscope. (In the aboveexample, the endoscope would be redirected to focus on the moving toolrather than on the tool approaching dangerously close to the organ).

It should be emphasized that all of the above (and the followingdisclosure) is enabled by constantly monitoring and locating/identifyingthe 3D spatial location of each element/tool in the surgicalenvironment.

The identification is provided by conventional means known to anyskilled in the art (e.g., image processing, optical means etc.).

According to some embodiments, the surgical tracking subsystemcomprises:

-   -   a. at least one endoscope adapted to acquire real-time images of        a surgical environment within the human body;    -   b. a maneuvering subsystem adapted to control the spatial        position of the endoscope during the laparoscopic surgery; and,    -   c. a tracking subsystem in communication with the maneuvering        subsystem, adapted to control the maneuvering subsystem so as to        direct and modify the spatial position of the endoscope to a        region of interest.

According to this embodiment, the tracking subsystem comprises a dataprocessor. The data processor is adapted to perform real-time imageprocessing of the surgical environment and to instruct the maneuveringsubsystem to modify the spatial position of the endoscope according toinput received from a maneuvering function f(t); the maneuveringfunction f(t) is adapted to (a) receive input from at least twoinstructing functions g_(i)(t), where i is 1, . . . , n and n≥2 andwhere t is time; i and n are integers; and (b) to output instructions tothe maneuvering subsystem based on the input from the at least twoinstructing functions g_(i)(t), so as to spatially position theendoscope to the region of interest.

According to one embodiment, the tool detection function MO is adaptedto detect tools in the surgical environment. According to thisembodiment, the tool detection function is adapted to detect surgicaltools in the surgical environment and to output instructions to thetracking subsystem to instruct the maneuvering subsystem to direct theendoscope to the detected surgical tools.

According to some embodiments, the functions g(t) may rank the differentdetected areas in the surgical environment according to a ranking scale(e.g., from 1 to 10) in which prohibited areas (i.e., areas which aredefined as area to which the surgical tools are forbidden to ‘enter)receive the lowest score (e.g., 1) and preferred areas (i.e., areaswhich are defined as area in which the surgical tools should bemaintained) receive the highest score (e.g., 10).

According to a preferred embodiment, one function MO is adapted todetect tools in the surgical environment and inform the maneuveringfunction f(t) if they are in preferred areas or in prohibited areas.

According to some embodiments, the movement detection function g₂(t)comprises a communicable database comprising the real-time 3D spatialpositions of each of the surgical tools in the surgical environment;means to detect movement of the at least one surgical tool when a changein the 3D spatial positions is received, and means to outputinstructions to the tracking subsystem to instruct the maneuveringsubsystem to direct the endoscope to the moved surgical tool.

According to some embodiments, the organ detection function g₃(t) isadapted to detect physiological organs in the surgical environment andto classify the detected organs as prohibited areas or preferred areas.For example, if the operator instructs the system that the specificsurgery is kidney surgery, the organ detection function g₃(t) willclassify the kidneys (or one kidney, if the surgery is specified to beon a single kidney) as a preferred area and other organs will beclassified as prohibited areas. According to some embodiments, the organdetection function is adapted to detect organs in the surgicalenvironment and to output instructions to the tracking subsystem toinstruct the maneuvering subsystem to direct the endoscope to thedetected organs. According to some embodiments, the right tool functionis adapted to detect surgical tool positioned to right of the endoscopeand to output instructions to the tracking subsystem to instruct themaneuvering system to constantly direct the endoscope on the right tooland to track the right tool.

According to some embodiments, the left tool function is adapted todetect surgical tool positioned to left of the endoscope and to outputinstructions to the tracking subsystem to instruct the maneuveringsystem to constantly direct the endoscope on the left tool and to trackthe left tool.

According to some embodiments, the collision detection function g₄(t) isadapted to detect prohibited areas within the surgical environment so asto prevent collisions between the endoscope and the prohibited areas.For example, if the endoscope is located in a narrow area in which aprecise movement of the same is preferred, the collision detectionfunction g₄(t) will detect and classify different areas (e.g., nerves,veins, walls of organs) as prohibited areas. Thus, according to thisembodiment, the collision prevention function is adapted to define apredetermined distance between the at least one surgical tool and ananatomical element within the surgical environment; and to outputinstructions to the tracking subsystem to instruct the maneuveringsubsystem to direct the endoscope to the surgical tool and theanatomical element within the surgical environment if the distancebetween the at least one surgical tool and an anatomical element is lessthan the predetermined distance. According to one embodiment of thepresent invention the anatomical element is selected from a groupconsisting of tissue, organ, another surgical tool and any combinationthereof.

According to some embodiments, the operator input function g₅(t) isadapted to receive an input from the operator. The input can be, forexample: an input regarding prohibited areas in the surgicalenvironment, an input regarding allowed areas in the surgicalenvironment, or an input regarding the region of interest and anycombination thereof. The operator input function g₅(t) can receiveinstructions from the operator before or during the surgery, and respondaccordingly. According to some embodiments, the operator input functionmay further comprise a selection algorithm for selection of areasselected from a group consisting of: prohibited areas, allowed areas,regions of interest, and any combination thereof. The selection may beperformed via an input device (e.g., a touch screen).

According to some embodiments, the operator input function g₅(t)comprises a communicable database; the communicable database is adaptedto receive an input from the operator of the system; the inputcomprising n 3D spatial positions; n is an integer greater than or equalto 2; and to output instructions to the tracking subsystem to instructthe maneuvering subsystem to direct the endoscope to the at least one 3Dspatial position received.

According to some embodiments, the prediction function g₆(t) is adaptedto provide data regarding a surgical environment at a time t_(f)> to,wherein to is the present time and t_(f) is a future time. Theprediction function g₆(t) may communicate with a database which storesdata regarding the environment of the surgery (e.g., the organs in theenvironment). This data may be used by the prediction function g₆(t) forthe prediction of expected or unexpected events or expected orunexpected objects during the operation. Thus, according to thisembodiment, the prediction function g₆(t) comprises a communicabledatabase storing each 3D spatial position of each of surgical toolwithin the surgical environment, such that each movement of eachsurgical tool is stored; the prediction function is adapted to (a) topredict the future 3D spatial position of each of the surgical tools (oreach object); and, (b) to output instructions to the tracking subsystemto instruct the maneuvering subsystem to direct the endoscope to thefuture 3D spatial position.

According to some embodiments, the past statistical analysis functiong₇(t) is adapted to provide data regarding the surgical environment orthe laparoscopic surgery based on past statistical data stored in adatabase. The data regarding the surgical environment may be forexample: data regarding prohibited areas, data regarding allowed areas,data regarding the region of interest and any combination thereof. Thus,according to this embodiment, the past statistical analysis functiong₆(t) comprises a communicable database storing each 3D spatial positionof each of surgical tool within the surgical environment, such that eachmovement of each surgical tool is stored; the past statistical analysisfunction g₆(t) is adapted to (a) perform statistical analysis on the 3Dspatial positions of each of the surgical tools in the past; and, (b) topredict the future 3D spatial position of each of the surgical tools;and, (c) to output instructions to the tracking subsystem to instructthe maneuvering subsystem to direct the endoscope to the future 3Dspatial position. Thus, according to the past statistical analysisfunction g₇(t), the past movements of each tool are analyzed and,according to this analysis, a prediction of the tool's next move isprovided.

According to some embodiments, the most used tool function g₈(t)comprises a communicable database counting the amount of movement ofeach surgical tool located within the surgical environment; the mostused tool function is adapted to output instructions to the trackingsubsystem to instruct the maneuvering subsystem to direct the endoscopeto constantly position the endoscope to track the movement of the mostmoved surgical tool. The amount of movement of a tool can be defined asthe total number of movements of that tool or the total distance thetool has moved.

According to some embodiments, the right tool function g₉(t) is adaptedto detect at least one surgical tool in a specified position in relationto the endoscope, preferably positioned to right of the endoscope and tooutput instructions to the tracking subsystem to instruct themaneuvering subsystem to constantly direct the endoscope to the righttool and to track the same. According to preferred embodiments, theright tool is defined as the tool positioned to the right of theendoscope; according to some embodiments, any tool can be defined as theright tool.

According to some embodiments, the left tool function g₁₀(t) is adaptedto detect at least one surgical tool in a specified position in relationto the endoscope, preferably positioned to left of the endoscope and tooutput instructions to the tracking subsystem to instruct themaneuvering subsystem to constantly direct the endoscope to the lefttool and to track the same. According to preferred embodiments, the lefttool is defined as the tool positioned to the left of the endoscope;according to other embodiments, any tool can be defined as the lefttool.

According to some embodiments, the field of view function g₁₁(t)comprises a communicable database comprising n 3D spatial positions; nis an integer greater than or equal to 2; the combination of all of then 3D spatial positions provides a predetermined field of view; the fieldof view function is adapted to output instructions to the trackingsubsystem to instruct the maneuvering subsystem to direct the endoscopeto at least one 3D spatial position substantially within then 3D spatialpositions so as to maintain a constant field of view.

According to some embodiments, the preferred volume zone function g₁₂(t)comprises a communicable database comprising n 3D spatial positions; nis an integer greater than or equal to 2; the n 3D spatial positionsprovide the preferred volume zone; the preferred volume zone functiong₁₂(t) is adapted to output instructions to the tracking subsystem toinstruct the maneuvering subsystem to direct the endoscope to at leastone 3D spatial position substantially within the preferred volume zone.

According to some embodiments, the no fly zone function g₁₃(t) comprisesa communicable database comprising n 3D spatial positions; n is aninteger greater than or equal to 2; the n 3D spatial positions define apredetermined volume within the surgical environment; the no fly zonefunction g₁₃(t) is adapted to output instructions to the trackingsubsystem to instruct the maneuvering subsystem to direct the endoscopeto at least one 3D spatial position substantially different from all then 3D spatial positions.

According to some embodiments, the proximity function g₁₄(t) is adaptedto define a predetermined distance between at least two surgical tools;and to output instructions to the tracking subsystem to instruct themaneuvering subsystem to direct the endoscope to the two surgical toolsif the distance between the two surgical tools is less than or if it isgreater than the predetermined distance.

According to some embodiments, the proximity function g₁₄(t) is adaptedto define a predetermined angle between at least three surgical tools;and to output instructions to the tracking subsystem to instruct themaneuvering subsystem to direct the endoscope to the three surgicaltools if the angle between the two surgical tools is less than or if itis greater than the predetermined angle.

According to some embodiments, the preferred volume zone functioncomprises a communicable database comprising n 3D spatial positions; nis an integer greater than or equals to 2; the n 3D spatial positionsprovides the preferred volume zone; the preferred volume zone functionis adapted to output instructions to the tracking subsystem to instructthe maneuvering system to direct the endoscope to the preferred volumezone.

According to some embodiments, the field of view function comprises acommunicable database comprising n 3D spatial positions; n is an integergreater than or equals to 2; the combination of all of the n 3D spatialpositions provides a predetermined field of view; the field of viewfunction is adapted to output instructions to the tracking subsystem toinstruct the maneuvering system to direct the endoscope to at least one3D spatial position substantially within then 3D spatial positions so asto maintain a constant field of view.

According to some embodiments, the no fly zone function comprises acommunicable database comprising n 3D spatial positions; n is an integergreater than or equals to 2; the n 3D spatial positions define apredetermined volume within the surgical environment; the no fly zonefunction is adapted to output instructions to the tracking subsystem toinstruct the maneuvering system to direct the endoscope to at least one3D spatial position substantially different from all the n 3D spatialpositions.

According to some embodiments, the most used tool function comprises acommunicable database counting the amount of movement of each surgicaltool located within the surgical environment; the most used toolfunction is adapted to output instructions to the tracking subsystem toinstruct the maneuvering system to direct the endoscope to constantlyposition the endoscope to track the movement of the most moved surgicaltool.

According to some embodiments, the prediction function g₆(t) is adaptedto provide data regarding a surgical environment in a time t_(f)>t,wherein t is the present time and t_(f) is the future time. Theprediction function g₆(t) may communicate with a database which storesdata regarding the environment of the surgery (e.g., the organs in theenvironment). This data may be used by the prediction function g₆(t) forthe prediction of expected or unexpected events or object during theoperation. Thus, according to this embodiment, the prediction functioncomprises a communicable database storing each 3D spatial position ofeach of surgical tool within the surgical environment, such that eachmovement of each surgical tool is stored; the prediction function isadapted to (a) to predict the future 3D spatial position of each of thesurgical tools; and, (b) to output instructions to the trackingsubsystem to instruct the maneuvering system to direct the endoscope tothe future 3D spatial position.

According to some embodiments, the past statistical analysis functiong₇(t) is adapted to provide data regarding the surgical environment orthe laparoscopic surgery based on past statistical data stored in adatabase. The data regarding the surgical environment may be forexample: data regarding prohibited areas, data regarding allowed areas,data regarding the region of interest. Thus, according to thisembodiment, the past statistical analysis function comprises acommunicable database storing each 3D spatial position of each ofsurgical tool within the surgical environment, such that each movementof each surgical tool is stored; the past statistical analysis functionis adapted to (a) statistical analyze the 3D spatial positions of eachof the surgical tools in the past; and, (b) to predict the future 3Dspatial position of each of the surgical tools; and, (c) to outputinstructions to the tracking subsystem to instruct the maneuveringsystem to direct the endoscope to the future 3D spatial position. Thus,according to the past statistical analysis function g₇(t), the pastmovements of each tool are analyzed and according to this analysis afuture prediction of the tool's next move is provided.

According to some embodiments, preferred tool function comprises acommunicable database, the database stores a preferred tool; thepreferred tool function is adapted to output instructions to thetracking subsystem to instruct the maneuvering system to constantlydirect the endoscope to the preferred tool, such that said endoscopeconstantly tracks said preferred tool.

Thus, according to the preferred tool function the endoscope constantlytracks the preferred tool, such that the field of view, as seen from theendoscope, is constantly maintained on said preferred tool. It should benoted that the user may define in said preferred tool function toconstantly track the tip of said preferred tool or alternatively, theuser may define in said preferred tool function to constantly track thebody or any location on the preferred tool.

According to some embodiments, the tagged tool function g₁₅(t) comprisesmeans adapted to tag at least one surgical tool within the surgicalenvironment and to output instructions to the tracking subsystem toinstruct the maneuvering subsystem to constantly direct the endoscope tothe tagged surgical tool. Thus, according to the tagged tool function,the endoscope constantly tracks the preferred (i.e., tagged) tool, suchthat the field of view, as seen from the endoscope, is constantlymaintained on the preferred (tagged) tool. It should be noted that theuser can define the tagged tool function to constantly track the tip ofthe preferred (tagged) tool, the body of the preferred (tagged) tool, orany other location on the preferred (tagged) tool.

According to some embodiments, the means are adapted to constantly tagat least one surgical tool within the surgical environment.

According to some embodiments, the preferred tool function g₁₆(t)comprises a communicable database. The database stores a preferred tool;and the preferred tool function is adapted to output instructions to thetracking subsystem to instruct the maneuvering subsystem to direct theendoscope to the preferred tool.

According to some embodiments, the system further comprises meansadapted to re-tag the at least one of the surgical tools until a desiredtool is selected.

According to some embodiments, the system further comprises meansadapted to toggle the surgical tools. According to some embodiments, thetoggling is performed manually or automatically.

According to some embodiments of the present invention, the weightingfunctions α_(i)(t) are time-varying functions (or constants), the valueof which is determined by the operator or the output of the instructingfunctions g_(i)(t). For example, if a specific function g_(i)(t)detected an important event or object, its weighting functions α_(i)(t)may be adjusted in order to elevate the chances that the maneuveringfunction f(t) will instruct the maneuvering subsystem to move theendoscope towards this important event or object.

According to some embodiments of the present invention, the trackingsubsystem may implement various image processing algorithms which mayalso be algorithms that are well known in the art. The image processingalgorithms may be for example: image stabilization algorithms, imageimprovement algorithms, image compilation algorithms, image enhancementalgorithms, image detection algorithms, image classification algorithms,image correlations with the cardiac cycle or the respiratory cycle ofthe human body, smoke reduction algorithms, vapor reduction algorithms,steam reduction algorithms and any combination thereof. Smoke, vapor andsteam reduction algorithms may be needed as it is known that, undercertain conditions, smoke, vapor or steam may be emitted by or from theendoscope. The image processing algorithm may also be implemented andused to analyze 2D or 3D representations which may be rendered from thereal-time images of the surgical environment.

According to some embodiments, the endoscope may comprise an imageacquisition device selected from a group consisting of: a camera, avideo camera, an electromagnetic sensor, a computer tomography imagingdevice, a fluoroscopic imaging device, an ultrasound imaging device, andany combination thereof.

According to some embodiments, the system may also comprise a displayadapted to provide input or output to the operator regarding theoperation of the system. The display may be used to output the acquiredreal-time images of a surgical environment with augmented realityelements. The display may also be used for the definition of the regionof interest by the operator.

According to some embodiments, the endoscope may be controlled be anendoscope controller for performing operations such as: acquiring thereal-time images and zooming-in to a predetermined area. For example,the endoscope controller may cause the endoscope to acquire thereal-time images in correlation with the cardiac cycle or therespiratory cycle of a human body.

According to some embodiments, the data processor of the presentinvention may operate a pattern recognition algorithm for assisting theoperation of the instructing functions g_(i)(t). The pattern recognitionalgorithm may be used as part of the image processing algorithm.

It should be emphasized that all of the above (and the followingdisclosure) is enabled by constantly monitoring and locating/identifyingthe 3D spatial location of each element/tool in the surgicalenvironment.

The identification is provided by conventional means known to anyskilled in the art (e.g., image processing, optical means etc.).

It should be emphasized that all of the above (and the followingdisclosure) is enabled by constantly monitoring and locating/identifyingthe 3D spatial location of each element/tool in the surgicalenvironment.

The identification is provided by conventional means known to anyskilled in the art (e.g., image processing, optical means etc.).

Reference is made now to FIG. 28 , which is a general schematic view ofan embodiment of a surgical tracking system 100. In this figure areillustrated surgical instruments 17 b and 17 c and an endoscope 21 whichmay be maneuvered by means of maneuvering subsystem 19 according to theinstructions received from a tracking subsystem operable by computer 15.

According to one embodiment of the present invention as defined in theabove, the user may define the field of view function as constantlymonitoring at least one of surgical instruments 17 b and 17 c.

According to this embodiment, the surgical tracking system 100 may alsocomprise one or more button operated wireless transmitters 12 a, whichtransmit, upon activation, a single code wave 14 through aerial 13 toconnected receiver 11 that produces a signal processed by computer 15,thereby directing and modifying the spatial position of endoscope 21 tothe region of interest, as defined by the field of view function.

Alternatively, according to the proximity rule, if the distance betweenthe surgical instruments 17 b and 17 c is smaller than a predetermineddistance (as defined by the collision prevention rule), the systemalerts the user that any movement of either one of the surgicalinstruments 17 b and 17 c that will reduce the distance is a RESTRICTEDmovement.

FIGS. 29 a-b show an embodiment wherein the fine control means is acontrol mechanism (1830) which attaches to the endoscope (1810). Thefine control mechanism attaches to the manual controls (1820) for thearticulating endoscope via a connector (1840). In a preferredembodiment, the connector can connect any endoscope control means withany articulating endoscope. FIG. 29 a shows the fine control mechanism(1830) before it is attached to the articulating endoscope (1810), whileFIG. 29 b shows the control mechanism (1830) attached to thearticulating endoscope (1810), with the endoscope manual control (1840)connected to the fine control mechanism via the connector (1830).

In some embodiments, such as that shown in FIG. 29 , hardware control ofthe articulation is used, with the current system in effect replacingthe surgeon by moving the controls of the articulating tool. In someembodiments, software control is used, with the current system in effectreplacing the controls of the articulating tool so that the toolarticulates based on commands coming directly from the current systemrather than via the tool's manual controls.

FIG. 30 shows an embodiment of the articulating endoscope (1810) in use.The endoscope (1810) is attached to the zoom mechanism of the coarsecontrol system (1960), which is attached to the articulating arm (1970)of the coarse control system. The fine control mechanism (1830) isattached to the articulating endoscope (1810) and also enabled to becontrolled (either in a wired manner or wirelessly) either automaticallyby the control system or manually by the endoscope operator. The finecontrol mechanism (1840) is also connected to the manual controls (1922,1924) of the articulating endoscope. In this example, one control (1922)is forward and one (1924) is backward, turning the endoscope tip (1950)toward the right of the figure.

FIG. 31 a-d shows articulation of an embodiment of the articulatingendoscope. FIG. 31 a illustrates the flexibility of the articulatingtip, showing it in typical positions—bent forwards, out of the plane ofthe paper (1952), to the right (1954), downward (1956), and to the leftand backward, into the plane of the paper (1958).

FIGS. 31 b-d illustrate the articulating tip (1950) in use, followingthe movements of the tip (2082) of a medical instrument (2080). In FIG.31 b , the endoscope tip (1950) is straight; it is not yet following thetip of the instrument (2082). In FIG. 31 c , the instrument tip (2082)has moved to the right and the tip of the endoscope (1950) has turnedright to follow the tip (2082) of the instrument. It can be seen fromthe angle of the endoscope (1950) that the pivoting point of theendoscope has not changed, although the field of view of the endoscope(1950) has changed significantly. In FIG. 31 d , the instrument tip(2082) has moved towards the endoscope and forward, out of the plane ofthe paper. The tip of the endoscope (1950) has rotated to follow themovement of the instrument tip (2082), but the pivoting point of theendoscope has not changed. It is clear from FIGS. 31 a-31 d that use ofthe articulating endoscope allows the surgeon a much larger field ofview than would be possible with only movement of an endoscope aroundthe pivoting point. Use of an articulating endoscope also minimizesmovement of the whole endoscope relative to the pivoting point, whichhas the possibility of causing unwanted movement of the pivoting pointand, therefore, unwanted movement of the field of view.

EXAMPLES

Examples are given in order to demonstrate embodiments of presentinvention. The examples describe the manner and process of embodimentsof the present invention and set forth the best mode contemplated by theinventors for carrying out the invention, but are not to be construed aslimiting the invention.

In the examples below, similar numbers refer to similar parts in all ofthe figures.

In FIGS. 32-45 in the examples below, for simplicity and clarity, arigid tool has been illustrated although the rules are equallyapplicable to both rigid and articulating tools.

Example 1—Tracking System with Collision Avoidance System

One embodiment of such a rule-based system will comprise the followingset of commands:

Detection (denoted by Gd):

Gd1 Tool location detection function

Gd2 Organ (e.g. Liver) detection function

Gd3 Movement (vector) calculation and estimation function

Gd4 Collision probability detection function

Tool Instructions (denoted Gt):

Gt1 Move according to manual command

Gt2 Stop movement

The scenario—manual move command by the surgeon:

Locations Gd1(t) and Gd2(t) are calculated in real time at each timestep (from an image or location marker).

Tool movement vector Gd3(t) is calculated from Gd1(t) as the differencebetween the current location and at least one previous location(probably also taking into account previous movement vectors).

The probability of collision—Gd4(t)—is calculated, for example, from thedifference between location Gd1 and location Gd2 (the smaller thedistance, the closer the proximity and the higher the probability ofcollision), from movement vector Gd3(t) indicating a collision, etc.

Tool Instructions Gt1 Weight function α₁(t)=1 If Gt1(t)<a predeterminedthreshold and 0 otherwise.

Tool Instructions Gt2 Weight function α₂(t)=1 If Gt2(t)>a predeterminedthreshold and 0 otherwise.

Tool Instructions=α₁(t)*Gt1+α₂(t)*Gt2(t).

In the examples given hereinbelow, the liver will be used as anexemplary organ. The rules can be applied in a similar manner to anyidentifiable organ or structure in the body.

In reference to FIG. 32 , which shows, in a non-limiting manner, anembodiment of a tracking system and collision avoidance system. Thesystem tracks a tool 310 and an organ, in this case a liver 320, inorder to determine whether a collision between the tool 310 and theorgan 320 is possible within the next time step. FIGS. 32 a and 32 bshow how the behavior of the system depends on the distance 330 betweenthe tool 310 and the organ 320, while FIGS. 32 c and 32 d show howmovement of the tool 310 affects the behavior. In FIG. 32 a , thedistance 330 between the tool 310 and the organ 320 is large enough thata collision is not possible in that time step. Since no collision ispossible, no movement of the tool is commanded. In FIG. 32 b , thedistance 330 between the tool 310 and the organ 320 is small enough thata collision is likely. In the embodiment illustrated, a movement 340 iscommanded to move the tool 310 away from the organ 320. In someembodiments, the system prevents movement 350, but does not commandmovement 340; in such embodiments, the tool 310 will remain close to theorgan 320. In some embodiments, the system warns/signals the operatorthat the move is RESTRICTED, but does not restrict movement 350 orcommand movement 340 away from the organ (320). Such a warning/signalingcan be visual or aural, using any of the methods known in the art.

FIGS. 32 c and 32 d illustrate schematically the effect of the movementof tool 310 on the collision avoidance system. In FIGS. 32 c and 32 d ,the tool 310 is close enough to the organ 320 that a collision betweenthe two is possible. If the system tracked only the positions of thetool 310 and the organ 320, then motion of the tool 310 away from theorgan 320 would be commanded. FIG. 32 c illustrates the effect of amovement 350 that would increase the distance between tool 310 and organ320. Since the movement 350 is away from organ 320, no collision ispossible in this time step and no movement of the tool 310 is commanded.

In FIG. 32 d , tool 310 is the same distance from organ 320 as in FIG.32 c . However, in FIG. 32 d , the movement 350 of the tool 310 istoward the organ 320, making a collision between tool 310 and organ 320possible. In some embodiments, a movement 340 is commanded to move thetool 310 away from the organ 320. In some embodiments, the systemprevents movement 350, but does not command movement 340; in thisembodiment the tool 310 will remain close to the organ 320. In someembodiments, the system warns the operator that move is RESTRICTED, butdoes not restrict movement 350 or command movement 340 away from theorgan (320). Such a warning can be visual or aural, using any of themethods known in the art.

As a non-limiting example, in an operation on the liver, the collisiondetection function can warn the operator that a collision between a tooland the liver is likely but not prevent the collision; in an operationon the liver, a collision between the tool and the liver may be desiredin order to remove tissue therefrom.

In another non-limiting example, in an operation on the gall bladder,the collision detection function can prevent a collision between thetool and the liver, either by preventing the movement or by commanding amovement redirecting the tool away from the liver.

Example 2—Tracking System with Soft Control—Fast Movement when Nothingis Nearby, Slow Movement when Something is Close

One embodiment of such rule-based system comprises the following set ofcommands:

Detection (denoted by Gd):

Main Tool location detection function (denoted by GdM);

Gd-tool1-K—Tool location detection function;

Gd-organ2-L—Organ (e.g. Liver) detection function;

Gd3 Main Tool Movement (vector) calculation and estimation function;

Gd4 Proximity probability detection function;

Tool Instructions (denoted Gt):

Gt1 Movement vector (direction and speed) according to manual command

The scenario—manual move command by the surgeon:

Locations GdM(t), Gd-tool1-K(t) and Gd-organ2-L(t) are calculated inreal time at each time step (from image or location marker).

Main Tool Movement Vector Gd3(t) is calculated per GdM (t) as thedifference between the current location and at least one previouslocation (probably also taking into account previous movement vectors)

The proximity of the main tool to other tools—Gd4(t)—is calculated, forexample, as the smallest of the differences between the main toollocation and the other tools' locations.

Tool Instructions Gt1 Weight function α₁(t) is proportional to toolproximity function Gd4(t), the closer the tool the slower the movementso that, for example

α₂(t)=Gd4/maximum(Gd4)

or

α₂(t)=log(Gd4/maximum(Gd4))

where maximum(Gd4) is the maximum distance which is likely to result ina collision given the distances, the speed of the tool and the movementvector.

Tool Instructions=α_(i)(t)*Gt1.

Example 3—Tracking System with No-Fly Rule/Function

In reference to FIG. 33 a-d , which shows, in a non-limiting manner, anembodiment of a tracking system with no-fly rule. The system tracks atool 310 with respect to a no-fly zone (460), in order to determinewhether the tool will enter the no-fly zone (460) within the next timestep. In this example, the no-fly zone 460 surrounds the liver.

FIGS. 33 a and 33 b show how the behavior of the system depends on thelocation of the tool tip with respect to the no-fly zone, while FIGS. 33c and 33 d show how movement of the tool affects the behavior.

In FIG. 33 a , the tool 310 is outside the no-fly zone rule/function 460and no movement of the tool is commanded. In FIG. 33 b , the tool 310 isinside the no-fly zone 460.

The no-fly zone rule/function performs as follows:

In the embodiment illustrated, a movement 350 is commanded to move thetool 310 away from the no-fly zone 460. In some embodiments, the systemprevents movement further into the no-fly zone (refers as movement 340,see FIG. 33 c ), but does not command movement 340; in such embodiments,the tool 310 will remain close to the no-fly zone 460.

In some embodiments, the system warns/signals the operator that the moveis RESTRICTED, but does not restrict movement further into the no-flyzone or command movement 340 away from the no-fly zone 460. Such awarning/signaling can be visual or aural, using any of the methods knownin the art.

FIGS. 33 c and 33 d illustrate schematically the effect of the tool'smovement on operation of the no-fly zone rule/function. In FIGS. 33 cand 33 d , the tool 310 is close enough to the no-fly zone 460 (distance330 is small enough) that it is possible for the tool to enter theno-fly zone during the next time step. FIG. 33 c illustrates the effectof a movement 340 that would increase the distance between tool 310 andno-fly zone 460. Since the movement 340 is away from no-fly zone 460, nocollision is possible in this time step and no movement of the tool 310is commanded.

In FIG. 33 d , tool 310 is the same distance from no-fly zone 460 as inFIG. 33 c . However, in FIG. 33 d , the movement 340 of the tool istoward no-fly zone 460, making it possible for tool 310 to enter no-flyzone 460. In the embodiment illustrated, a movement 350 is commanded tomove the tool 310 away from the no-fly zone 460. In some embodiments,the system prevents movement 340, but does not command movement 350; insuch embodiments, the tool 310 will remain close to the no-fly zone 460.In some embodiments, the system warns/signals the operator that the moveis RESTRICTED, but does not restrict movement 340 or command movement350 away from the no-fly zone rule/function 460. Such awarning/signaling can be visual or aural, using any of the methods knownin the art.

Example 4—Tracking System with Preferred Volume Zone Rule/Function

In reference to FIG. 34 a-d , which shows, in a non-limiting manner, anembodiment of a tracking system with a preferred volume zonefunction/rule.

The system tracks a tool 310 with respect to a preferred volume zone(570), in order to determine whether the tool will leave the preferredvolume (570) within the next time step.

In this example, the preferred volume zone 570 extends over the rightlobe of the liver. FIGS. 34 a and 34 b show how the behavior of thesystem depends on the location of the tool tip with respect to thepreferred volume zone 570, while FIGS. 34 c and 34 d show how movementof the tool affects the behavior (i.e., the preferred volume zonerule/function).

In FIG. 34 a , the tool 310 is inside the preferred volume zone 570 andno movement of the tool is commanded. In FIG. 34 b , the tool 310 isoutside the preferred volume zone 570.

In the embodiment illustrated, a movement 340 is commanded to move thetool 310 away from the preferred volume zone 570. In some embodiments,the system prevents movement 340; in such embodiments, the tool 310 willremain close to the preferred volume zone 570. In some embodiments, thesystem warns/signals the operator that the move 340 is RESTRICTED. Sucha warning/signaling can be visual or aural, using any of the methodsknown in the art.

FIGS. 34 c and 34 d illustrate schematically the effect of the tool'smovement on operation of the preferred volume rule/function. In FIGS. 34c and 34 d , the tool 310 is close enough to the edge of preferredvolume zone 570 that it is possible for the tool to leave the preferredvolume zone during the next time step.

FIG. 34 c illustrates the effect of a movement 350 that would take thetool 310 deeper into preferred volume zone 570. Since the movement 350is into preferred volume 570, said movement is an allowed movement.

In FIG. 34 d , the movement 350 of the tool is out of the preferredvolume 570, making it possible for tool 310 to leave preferred volume570.

According to one embodiment illustrated, a movement 340 is commanded tomove the tool 310 into the preferred volume zone 570. In someembodiments, the system prevents movement 350, but does not commandmovement 340; in such embodiments, the tool 310 will remain close to thepreferred volume zone 570. In some embodiments, the system warns/signalsthe operator that the move is RESTRICTED, but does not restrict movement350 or command movement 340 away from the preferred volume zone 570.Such a warning/signaling can be visual or aural, using any of themethods known in the art.

Example 5—Organ/Tool Detection Function

In reference to FIG. 35 , which shows, in a non-limiting manner, anembodiment of an organ detection system (however, it should be notedthat the same is provided for detection of tools, instead of organs).

For each organ, the 3D spatial positions of the organs stored in adatabase. In FIG. 35 , the perimeter of each organ is marked, toindicate the edge of the volume of 3D spatial locations stored in thedatabase.

In FIG. 35 , the liver 610 is labeled with a dashed line. The stomach620 is labeled with a long-dashed line, the intestine 630 with a solidline and the gall bladder 640 is labeled with a dotted line.

In some embodiments, a label or tag visible to the operator is alsopresented. Any method of displaying identifying markers known in the artcan be used. For non-limiting example, in an enhanced display, coloredor patterned markers can indicate the locations of the organs, with themarker either indicating the perimeter of the organ or the area of thedisplay in which it appears.

Example 6—Tool Detection Function

In reference to FIG. 36 , which shows, in a non-limiting manner, anembodiment of a tool detection function. For each tool, the 3D spatialpositions of the tools stored in a database. In FIG. 36 , the perimeterof each tool is marked, to indicate the edge of the volume of 3D spatiallocations stored in the database. In FIG. 36 , the left tool is labeledwith a dashed line while the right tool is labeled with a dotted line.

In some embodiments, a label or tag visible to the operator is alsopresented. Any method of displaying identifying markers known in the artcan be used. For non-limiting example, in an enhanced display, coloredor patterned markers can indicate the locations of the tools, with themarker either indicating the perimeter of the tool or the area of thedisplay in which it appears.

Example 7—Movement Detection Function/Rule

In reference to FIG. 37 a-b , which shows, in a non-limiting manner, anembodiment of a movement detection function/rule. FIG. 37 aschematically illustrates a liver 810, a left tool 820 and a right tool830 at a time t. FIG. 37 b schematically illustrates the liver 810, lefttool 820 and right tool 830 at a later time t+Δt, where Δt is a smalltime interval. In this example, the left tool 820 has moved downward(towards the direction of liver 810) in the time interval Δt.

The system has detected movement of left tool 820 and labels it. This isillustrated schematically in FIG. 37 b by a dashed line around left tool820.

Example 8—Prediction Function

In reference to FIG. 38 a-d , which shows, in a non-limiting manner, anembodiment of the above discussed prediction function.

FIG. 38 a shows a left tool 920 and a right tool 930 at a time t.

FIG. 38 b shows the same tools at a later time t+Δt, where Δt is a smalltime interval. Left tool 920 is moving to the right and downward, whileright tool 930 is moving to the left and upward. If the motion continues(shown by the dashed line in FIG. 38 c ), then by the end of the nexttime interval, in other words, at some time between time t+Δt and timet+2Δt, the tools will collide, as shown by tool tips within the dottedcircle 950 in FIG. 38 c.

In this embodiment, the system automatically prevents predictedcollisions and, in this example, the system applies a motion 940 toredirect left tool 920 so as to prevent the collision.

In some embodiments, the system warns/signals the operator that acollision is likely to occur, but does not alter the movement of anytool. Such a warning/signaling can be visual or aural, using any of themethods known in the art.

In some embodiments, the prediction function can be enabled to, fornon-limiting example, alter the field of view to follow the predictedmovement of a tool or of an organ, to warn of (or prevent) predictedmotion into a no-fly zone, to warn of (or prevent) predicted motion outof a preferred zone.

Example 9—Right Tool Function/Rule

In reference to FIG. 39 , which shows, in a non-limiting manner, anembodiment of a right tool function. FIG. 39 schematically illustrates aliver 1010, a left tool 1020 and a right tool 1030. The right tool,illustrated schematically by the dashed line 1040, is labeled and its 3Dspatial location is constantly and real-time stored in a database. Now,according to the right tool function/rule the endoscope constantlytracks the right tool.

It should be pointed out that the same rule/function applies for theleft tool (the left tool function/rule).

Example 10—Field of View Function/Rule

In reference to FIG. 40 a-b , which shows, in a non-limiting manner, anembodiment of a field of view function/rule.

FIG. 40 a schematically illustrates a field of view of the abdomen at atime t. In the field of view are the liver 1110, stomach 1120,intestines 1130 and gall bladder 1140.

The gall bladder is nearly completely visible at the left of the fieldof view. Two tools are also in the field of view, with their tips inproximity with the liver. These are left tool 1150 and right tool 1160.In this example, the field of view function/rule tracks left tool 1150.In this example, left tool 1150 is moving to the right, as indicated byarrow 1170.

FIG. 40 b shows the field of view at time t+Δt. The field of view hasmoved to the right so that the tip of left tool 1150 is still nearly atthe center of the field of view. It can be seen that much less of gallbladder 1140 is visible, while more of right tool 1160 has entered thefield of view.

The field of view function/rule can be set to follow a selected tool, asin this example or to keep a selected organ in the center of the fieldof view. It can also be set to keep a particular set of tools in thefield of view, zooming in or out as necessary to prevent any of thechosen tools from being outside the field of view.

Alternatively, the field of view function/rule defines n 3D spatialpositions; n is an integer greater than or equal to 2; the combinationof all of said n 3D spatial positions provides a predetermined field ofview.

Each movement of the endoscope or the surgical tool within said n 3Dspatial positions is an allowed movement and any movement of theendoscope or the surgical tool outside said n 3D spatial positions is arestricted movement.

Alternatively, said the field of view function/rule defines n 3D spatialpositions; n is an integer greater than or equal to 2; the combinationof all of said n 3D spatial positions provides a predetermined field ofview.

According to the field of view function/rule, the endoscope is relocatedif movement has been detected by said detection means, such that saidfield of view is maintained.

Example 11—Tagged Tool Function/Rule (or Alternatively the PreferredTool Rule)

In reference to FIG. 41 , which shows, in a non-limiting manner, anembodiment of a tagged tool function/rule.

FIG. 41 shows three tools (1220, 1230 and 1240) in proximity to theorgan of interest, in this example, the liver 1210.

The tool most of interest to the surgeon, at this point during theoperation, is tool 1240. Tool 1240 has been tagged (dotted line 1250);the 3D spatial location of tool 1240 is constantly stored in a databaseand this spatial location has been labeled as one of interest.

The system can use this tagging for many purposes, including, but notlimited to, keeping tool 1240 in the center of the field of view,predicting its future motion, keeping it from colliding with other toolsor keeping other tools from colliding with it, instructing the endoscopeto constantly monitor and track said tagged tool 1250 and so on.

It should be noted that in the preferred tool rule, the system tags oneof the tools and performs as in the tagged tool rule/function.

Example 12—Proximity Function/Rule

In reference to FIG. 42 a-c , which shows, in a non-limiting manner, anembodiment of a proximity function/rule.

FIG. 42 a schematically illustrates two tools (1310 and 1320) separatedby a distance 1330 which is greater than a predefined proximitydistance. Since tool 1310 is not within proximity of tool 1320, thefield of view (1380) does not move.

FIG. 42 b schematically illustrates two tools (1310 and 1320) separatedby a distance 1330 which is less than a predefined proximity distance.

Since tool 1310 is within proximity of tool 1320, the field of view 1380moves upward, illustrated schematically by arrow 1340, until the tips oftool 1310 and tool 1320 are in the center of field of view 1380 (FIG. 42c ).

Alternatively the once the distance 1330 between the two tool 1320 and1310 is smaller than a predetermined distance, the system alerts theuser of said proximity (which might lead to a collision between the twotools). Alternatively, the system moves one of the tools away from theother one.

Example 13—Operator Input Function/Rule

In reference to FIG. 43 a-b , which shows, in a non-limiting manner, anembodiment of an operator input function/rule. According to thisembodiment, input is received from the operator.

In the following example, the input received from the operator is whichtool to track.

FIG. 43 a schematically illustrates an endoscope with field of view 1480showing a liver 1410 and two tools 1420 and 1430. Operator 1450 firstselects the tip of the left tool as the region of interest, preferablyby touching the tool tip on the screen, causing the system to tag (1440)the tip of the left tool.

As illustrated in FIG. 43 b , the system then directs and modifies thespatial position of the endoscope so that the tagged tool tip 1440 is inthe center of the field of view 1480.

Another example of the operator input function/rule is the following:

If a tool has been moved closely to an organ in the surgicalenvironment, according to the proximity rule or the collision preventionrule, the system will, according to one embodiment, prevent the movementof the surgical tool.

According to one embodiment of the present invention, once the surgicaltool has been stopped, any movement of said tool in the direction isinterpreted as input from the operator to continue the movement of saidsurgical tool in said direction.

Thus, according to this embodiment, the operator input function/rulereceives input from the operator (i.e., physician) to continue the moveof said surgical tool (even though it is “against” the collisionprevention rule). Said input is simply in the form of the continuedmovement of the surgical tool (after the alert of the system or afterthe movement prevention by the system).

Example 14—Constant Field of View Rule/Function

In reference to FIGS. 44 a-d , which shows, in a non-limiting manner, anembodiment of a tracking system with a constant field of viewrule/function.

In many endoscopic systems, the tip lens in the camera optics is not ata right angle to the sides of the endoscope. Conventionally, the tiplens angle is described relative to the right angle, so that a tip lensat right angles to the sides of the endoscope is described as having anangle of 0. Typically, angled endoscope tip lenses have an angle of 30°or 45°. This tip lens angle affects the image seen during zooming. FIG.44 illustrates, in an out-of-scale manner, for a conventional system,the effect of zooming in the field of view in an endoscope with tip lensset straight in the end (FIGS. 44 a and 44 b ) vs. the effect of zoomingin the field of view in an endoscope with angled tip lens (FIGS. 44 cand 44 d ).

FIGS. 44 a and 44 c illustrate the endoscope (100), the object it isviewing (200) and the image seen by the endoscope camera (130) beforethe zoom. The solid arrows (160) show the limits of the FOV and thedashed arrow (170), the center of the field of view (FOV); since theobject is in the center of the FOV, an image of the object (210) is inthe center of the camera image (130). FIGS. 44 b and 44 d illustrate theendoscope (100), the object it is viewing (200) and the image seen bythe endoscope camera (130) after the zoom. The solid arrows (160) showthe limits of the FOV and the dashed arrow (170), the center of thefield of view.

If the tip lens is set straight in the end of the endoscope (FIGS. 44 aand 44 b ), an object (200) in the center of the field of view will bein the center of the field of view (FOV) (and the camera image) (130)both before (FIG. 44 a ) and after (FIG. 44 b ) the zoom. However, ifthe tip lens is set at an angle in the end of the endoscope (FIGS. 44 cand 44 d ), then an object that is in the center of the FOV (and thecamera image) before the zoom (FIG. 44 c ) will not be in the center ofthe FOV (or the camera image) after the zoom (FIG. 44 d ) since thedirection of motion of the endoscope is not the direction in which thecenter of the field of view (170) points.

In an embodiment of the system of the present invention, unlike inconventional systems, the controlling means maintains the center of thefield of view (FOV) during zoom independent of the tip lens angle. Anadvantage of controlling the zoom of the endoscope via a data processingsystem is that the tip lens angle does not need to be input to the dataprocessing system, obviating a possible source of error.

According to one embodiment of the present invention, the endoscope'smovement will be adjusted in order to maintain a constant field of view.

Example 15—Misalignment Rule/Function

According to some embodiments of the present invention, the system caninform the user of any misalignment of the same system.

Misalignment of the system may cause parasitic movement of the endoscopetip, where the endoscope tip does not move exactly in the expecteddirection. According to one embodiment of the system, the systemcomprises sensors (e.g., gyroscopes, accelerometers and any combinationthereof) that calculate/estimates the position of the pivot point inreal time in order to (a) inform the user of misalignment; or (b)calculate the misalignment so that the system can adjust its movement toprevent parasitic movement.

Example 16—Change of Speed Rule/Function

In reference to FIG. 45 , which shows, in a non-limiting manner, anembodiment of a tracking system with a change of speed rule/function.

In conventional endoscopic control systems, motion of the endoscopeoccurs at a single speed. This speed is fairly fast so that theendoscope can be moved rapidly between locations that are wellseparated. However, this means that making fine adjustments so difficultthat fine adjustments are normally not made. In an embodiment of thepresent invention, the speed of the tip of the endoscope isautomatically varied such that, the closer the endoscope tip is to anobject, be it a tool, an obstacle, or the object of interest, the moreslowly it moves. In this embodiment, as shown in FIG. 45 , measurementsare made of the distance X (150) from the tip (195) of the endoscope(100) to the pivot point of the endoscope (190), where said pivot pointis at or near the surface of the skin (1100) of a patient (1000).Measurements are also made of the distance Y (250) from the tip of theendoscope (195) to the object in the center of the scene of view (200).From a predetermined velocity V_(p), the actual velocity of the tip ofthe endoscope at a given time, V_(act), is calculated from

$V_{act} \propto {\frac{Y}{X}V_{p}}$

Therefore, the closer to the object at the center of the scene of view,the more slowly the endoscope moves, making it possible to use automaticcontrol of even fine adjustments, and reducing the probability that theendoscope will come in contact with tissue or instruments.

Example 17—Articulation of Tool

In reference to FIG. 46 a-b , a non-limiting example of movement of anarticulating tool (310), here an endoscope, is shown schematically.

In FIG. 46 a-b , the endoscope (310) is moved so that, instead ofviewing the outer side of the liver (320) from the right, it views theinner side of the liver (320) from the left.

FIG. 46 a shows the endoscope (310) at the beginning of the movement. Itis fully extended and its tip (318) is positioned about halfway up theouter side of the liver. The dashed line shows the movement of the base(312) of the endoscope, which will move in a straight line from itsstarting position (FIG. 46 a ) to its final position (FIG. 20 b ). Thedotted line shows the movement of the endoscope tip (318)—the tip (318)moves upward, over the top of the liver (320), and then down the innerside of the liver (320), to allow imaging of the left (inner) side ofthe liver from between the liver (320) and the lungs (1790).

In FIG. 46 b , the movement has been completed. The endoscope tip (318)now points rightward; the articulating section (316) being curved sothat the endoscope (310) views the right side of the liver (310), withthe endoscope tip (318) being between the liver (320) and the lungs(1790) while its base (312) remains on the right side of the body.

Example 18—Articulation of Tool

In reference to FIG. 47 , a non-limiting example of flexion of anarticulating tool (310), here an endoscope, is shown schematically.

In FIG. 47 , portions of the small intestine (1795) are shownschematically. The endoscope enters the body from the body's right side(body not shown), and views a portion of the small intestine (1795F)from the left and below. The articulating section of the endoscope (316)bypasses a loop of small intestine (1795A), passes between two portionsof small intestine (1795B, 1795C), and over other portions of smallintestine (1795D, 1795E) so that the endoscope's tip (318) views thedesired portion of the small intestine (1795F) from the desireddirection.

In the foregoing description, embodiments of the invention, includingpreferred embodiments, have been presented for the purpose ofillustration and description. They are not intended to be exhaustive orto limit the invention to the precise form disclosed. Obviousmodifications or variations are possible in light of the aboveteachings. The embodiments were chosen and described to provide the bestillustration of the principals of the invention and its practicalapplication, and to enable one of ordinary skill in the art to utilizethe invention in various embodiments and with various modifications asare suited to the particular use contemplated. All such modificationsand variations are within the scope of the invention as determined bythe appended claims when interpreted in accordance with the breadth theyare fairly, legally, and equitably entitled.

1. A method of using a structured-light based endoscope, comprising:capturing at least one real-time 2D image of at least a portion of saidfield of view using a camera of an endoscope; illuminating in real timeat least a portion of said at least one object within at least a portionof said field of view with a structured light pattern; detecting lightreflected from said field of view; from said light reflected from saidfield of view, generating said 3D image of said field of view andcalculating 3D locations of points on a surface of said object, whereinsaid 3D image is constructable from said detected light reflected fromsaid field of view and said structured light pattern; and real-timelocating the 3D spatial position at any given time t of at least one ofthe endoscope and a surgical tool; causing the system to generate a useralert if a distance between the surface and said at least one of theendoscope and surgical tool, as determined using the 3D spatialposition, falls below a predetermined distance.
 2. The method of claim1, wherein the structured light pattern comprises at least one time andspace varying predetermined light pattern.
 3. The method of claim 1,further comprising causing a maneuvering system to maneuver saidendoscope in at least two degrees of freedom.
 4. The method of claim 1,wherein said endoscope is an articulating endoscope.
 5. The method ofclaim 1, further comprising constructing said 3D image by calculatingworld coordinates of at least one point on said at least one object. 6.The method of claim 5, comprising calculating said world coordinates ofsaid at least one point on said at least one object from the followingequation:$\overset{\_}{\alpha} = {\frac{n^{T}{\overset{\sim}{x}}_{p}}{n^{T}R_{p}v_{c}}.}$where n^(T) is the transpose of the normal to the plane defined by thestripe ID x_(p), {tilde over (x)}_(p)=x_(p)+[δx_(p), 0, f_(p)]^(T) isthe perturbed stripe ID x_(p), R_(p) is the rotation matrix defining thetransformation between the world coordinate system and the projectorcoordinate system and v_(c) is the direction of the ray between thestripe ID and the object point.
 7. The method of claim 5, comprising,for any point X_(w) in world coordinate system, calculating thecoordinate X_(c) of the same point in the camera coordinate systemaccording to the following equation:X _(c) =C _(c) X _(w), where C_(c), the camera perspective projectionmatrix, is of the form $C_{c} = {{{\alpha\begin{bmatrix}f_{x} & {kf}_{y} & x_{c}^{0} \\0 & f_{y} & y_{c}^{0} \\0 & 0 & 1\end{bmatrix}}\begin{bmatrix}R_{c} & t_{c}\end{bmatrix}}.}$ where α is a proportion coefficient, fx and fy are thecamera focal length scaled to each of the camera image dimensions, k isthe shear of the camera coordinate system, x_(c) ⁰ and y_(c) ⁰ are theorigin of Xc in image coordinates, and R_(c) and t_(c) define thetransformation between the world coordinate system and the lightsource's coordinate system, with R_(c) being a rotation matrix and t_(c)a translation matrix.
 8. The method of claim 7, comprising definingx_(p) ⁰ to be the x-coordinate of the intersection of the optical axisand the projector.
 9. The method of claim 1, comprising, for any pointX_(w) in world coordinate system, calculating the coordinate X_(p) ofthe same point in the light source coordinate system according to thefollowing equation:Xp=CpXw, where Cp, the light source perspective projection matrix, is ofthe form $C_{p} = {{\alpha\begin{bmatrix}f_{p} & 0 & x_{p}^{0} \\0 & 0 & 1\end{bmatrix}}\begin{bmatrix}R_{p} & t_{p}\end{bmatrix}}$ where α is a proportion coefficient, f_(p) is the lightsource focal length scaled to projector dimensions, x_(p) ⁰ is theorigin of Xp in projector coordinates, and R_(p) and t_(p) define thetransformation between the world coordinate system and the lightsource's coordinate system, with R_(p) being a rotation matrix and t_(p)a translation matrix.
 10. The method of claim 1, comprising calculatingthe world coordinates p_(w) of a point P according to the followingequation: ${\begin{pmatrix}p_{p} \\p_{s}\end{pmatrix} - \begin{pmatrix}{F_{c}\left( {p_{w};\Theta_{c}} \right.} \\{F_{p}\left( {p_{w};\Theta_{p}} \right.}\end{pmatrix}} = 0$ where p_(p)=(x_(p),y_(p))^(t) is the pixelcoordinate of said point, p_(x)=(x_(s)) is the stripe value of saidpoint P, F_(c)(P_(w); Θ_(c))=P_(P)−ϵ_(p) is the noise-free value of thevector of pixel coordinates, where P_(p) is the vector of measured pixelcoordinates and ϵ_(p) is the vector of errors in the pixel coordinates;F_(p)(P_(w); Θ_(p))=P_(s)−ϵ_(s) is the noise-free value of the vector ofstripe coordinates, where P_(s) is the vector of measured stripecoordinates and ϵ_(s) is the vector of errors in the stripe coordinates.11. The method of claim 10, comprising estimating the world coordinatesp_(w) of a point P according to the following non-linear least squares(NLLS) equations:${\min\limits_{\Theta_{c}}{{P_{p} - {F_{c}\left( {P_{w};\Theta_{c}} \right.}}}^{2}}{\min\limits_{\Theta_{p}}{{P_{s} - {F_{p}\left( {P_{w};\Theta_{p}} \right.}}}^{2}}$12. The method of claim 11, comprising solving said NLLS equations usinga NLLS solving algorithm selected from the group consisting of theGauss-Newton technique, the quasi-Newton technique, and theLevenberg-Marquardt technique.
 13. The method of claim 1, comprisingcalculating the location, in world coordinates, of a kth point on theobject according to the following equation:$p_{w}^{k} = \frac{\begin{matrix}{C_{1,2,1}^{k} - {x_{p}C_{3,2,1}^{k}} - {y_{p}C_{1,3,1}^{k}} -} \\{{x_{s}C_{1,2,2}^{k}} + {x_{s}x_{p}C_{3,2,2}^{k}} + {x_{s}y_{p}C_{1,3,2}^{k}}}\end{matrix}}{\begin{matrix}{C_{1,2,1}^{4} - {x_{p}C_{3,2,1}^{4}} - {y_{p}C_{1,3,1}^{4}} -} \\{{x_{s}C_{1,2,2}^{4}} + {x_{s}x_{p}C_{3,2,2}^{k}} + {x_{s}y_{p}C_{1,3,2}^{4}}}\end{matrix}}$ where C_(i,j,l) ^(k)=det(C_(c) ^(i), C_(c) ^(j), C_(p)^(l), e_(k)) are constants which depend only on a camera perspectivetransformation matrix and a projector perspective transformation matrix.14. The method of claim 1, further comprising displaying said 3D imageon an image display.